Saturday, February 6, 2016

Fun with Bugs #41 - Bugs Fixed in MySQL 5.7.11

I've just noted that Oracle had released new versions of MySQL on February 5, 2016 formally, so while these days I am mostly thinking about the ways to do support properly, remembering my colleagues and trying to understand some of RocksDB internals, it's time to postpone all these and write about bugs again. This time about some of the public bug reports from MySQL Community and Oracle engineers that were fixed by Oracle in MySQL 5.7.11.

As usual, I'll try to mention who had reported a bug and who verified it, as I think that names matter in MySQL world. I'll concentrate mostly on InnoDB, replication and optimizer bug reports, trying to highlight regressions clearly.

As usual, I prefer to start with InnoDB bugs:
  • Bug #75112 - "Optimize table removes the data directory in partitions". This bug was reported by my former colleague Nilnandan Joshi and verified by Umesh. The fix should appear in MySQL 5.6.30 also.
  • Bug #79775 - "Innodb doesn't update index stats when dropping virtual column in virtual index". It was reported by Shaohua Wang and probably was verified and fixed by him. I am always happy to see Oracle engineers reporting MySQL bugs in public! Check also his Bug #79773, "handler::my_eval_gcolumn_expr doesn't set correct null bit" fixed in 5.7.11.
  • Bug #79691 - "Block Memcached from modifying tables that contain indexed virtual columns". One more  case of public bug reporting by Oracle engineer. This time it was Thirunarayanan Balathandayuthapani.
  • Bug #79516 - "Server crashes on invalid value for variable innodb_saved_page_number_debug". When I hit third public report by Oracle engineer, this time even about a crash, I started to think that there is a new generation of engineers there that just do right things from the very beginning. Thank you, Naga Satyanarayana Bodapati, for reporting this bug in public!
  • Bug #79030 - "REORGANIZE PARTITION MOVES PARTITION TO TABLE-LEVEL TABLESPACE BY DEFAULT". When I see all caps in the bug synopsis I suspect it's from Oracle engineer (and from Oracle's internal bugs database, originally). This bug was reported by Darshan M n.
  • Bug #79201 - "InnoDB: Too small buffer pool for innodb_page_size=64k". Marko Mäkelä had never hesitated to report bugs in public. Check also his Bug #79150, "InnoDB: Remove runtime checks for 32-bit file offsets". Thank you for noting and fixing these! 
  • Bug #79434 - "Incorrect NUMA-specific code in buf_chunk_init()". It was reported by my former colleague Alexey Kopytov and verified by Sinisa Milivojevic formally. Check also his Bug #79354, "Incomplete NUMA support on buffer pool resizing".
  • Bug #78960 - "Unclear note "InnoDB: not started" in the error log". I've reported this bug back in October, 2015, based on the log from some customer issue. The message that was produced during loading of the buffer pool looked scary and misleading for customer, so I reported the bug immediately. It was verified by Umesh and is also fixed in 5.8.0.
  • Bug #78955 - "Import Dump with InnoDB Fulltext Index and Constraint fails". This regression bug was reported by Nico Niepraschk and verified by Umesh.
  • Bug #78831 - "InnoDB looks for buffer pool file name in '/' if innodb_data_home_dir is empty". It was reported by someone named NOT_FOUND NOT_FOUND. I'd like to find out one day who is it, as I see other useful bug reports under the same name, all fixed.
  • Bug #78714 - "InnoDB: Failing assertion: !(&dict_sys->mutex)->is_owned() in 3049", was reported by my former colleague Roel Van de Paar and verified by Umesh
  • Bug #78647 - "POTENTIAL RUNNING OUT OF SPACE IN REDO - INVALID MARGIN CALCULATIONS". This bug was reported by Pawel Olchawa and probably verified by Marko Mäkelä. The fix was mostly to increase the minimum value of innodb_log_file_size from 1MB to 4MB.
  • Bug #78262 - "Adding indexes on empty table is slow with large innodb_sort_buffer_size.". It was reported by Jean-François Gagné (who had made a great presentation at FOSDEM 2016, by the way) and verified by Umesh.
  • Bug #74832 - "ut_delay missing compiler barrier, UT_RELAX_CPU isn't at all relaxing". This bug was reported long time ago by Stewart Smith, who had provided the patch also. It was formally processed by Umesh.
The only public bug in partitioning that is listed in the release notes as fixed is Bug #79145, by Debarun Banerjee, who is probably working for Oracle.

Now, let's move on to replication (and XA transactions) bugs:
  • Bug #79416 - "InnoDB: Failing assertion: total_trx >= trx_sys->n_prepared_trx". This (debug builds only) bug was reported by Roel Van de Paar and verified by Umesh. See also Bug #78695, "Assertion `static_cast<Sql_cmd_xa_commit*>(thd->lex->m_sql_cmd)-> get_xa_opt() =".
  • Bug #77740 - "silent failure to start if mysql.gtids_executed gets HA_ERR_LOCK_WAIT_TIMEOUT". It was reported and verified by Shane Bester. The fix is to NOT allow XA transactions to modify the mysql.gtid_executed table.
  • Bug #77417 - "Applying create temporary table SQL on a slave with replicate-rewrite-db fails". This bug was reported by JAMES HETHERINGTON (who seems to be a customer of Oracle) and verified by Umesh. Good to see Oracle customers following my old advice and reporting bugs in public!
  • Bug #77393 - "Failure to generate GTID leads to inconsistency". It was reported by Davi Arnaut, who had provided a patch, and verified by Umesh.
  • Bug #77354 - "multi drop temp table statements in UDF cause replication fails using 5.6 GTID". This bug was reported by Fungo Wang (check his other replication-related bug reports, still "Verified") and verified by Umesh (no wonder, he seems to verify up to 70% of all bugs recently...)
  • Bug #77288 - "Variables in alter event not replicated properly". It was reported by Trey Raymond and verified by Umesh.
  • Bug #77249 - "Drop table may cause slaves to break". This bug was reported by  Santosh Praneeth Banda (who is a very active bug reporter, I see 7 more replication bugs) and verified by Umesh. It is also fixed in MySQL 5.6.29.
  • Bug #71859 - "Replication broken after creation of scheduled events". It was reported by Shahid Hussain and verified by Sveta Smirnova long time ago. The bug is also fixed in MySQL 5.6.29.
Let me conclude with several optimizer-related bugs:
  • Bug #79590 - "ERROR 1093 (HY000): You can't specify target table '.' for update in FROM clause". It was reported by Zhang Jinzhu and verified by Miguel Solorzano.
  • Bug #79549 - "Regression: “unknown column” for outer computed values used inside a subquery". This bug was reported by Chris Adams and verified by Miguel Solorzano.
  • Bug #79398 - "view using hexadecimal or bit literal gives wrong results". It was reported, verified and probably fixed by Guilhem Bichot.
  • Bug #79333 - "Unexpected error 1093 on nested subquery for update". This regression bug was reported by Danielle McLean and verified by Miguel Solorzano
  • Bug #79294 - "Query with double nested subquery much slower in 5.7". This regression bug was reported by Jonathan Corwin and verified by Umesh. Nice to see it fixed.
  • Bug #79194 - "Wrong result on outer join with uncorrelated subquery, derived_merge". This regression bug was reported by Elena Stepanova and verified by Miguel Solorzano.
  • Bug #79131 - "Get negative FOUND_ROWS() for UNION stmt". It was reported by Su Dylan and verified by Umesh. Based on the comments, it's also a regression bug in 5.7 comparing to 5.6, even though it is not tagged like that.
  • Bug #78946 - "WRONG RESULT FOR SELECT NULL IN (<SUBQUERY>) ". One more bug report from an Oracle developer, Dag Wanvik. Good that he had found, reported and probably fixed this regression, but in general I am negatively surprised by the number of recent optimizer regression bugs in MySQL 5.7 that are found only after they are introduced into public releases.
There are many more bug fixes, changes in behavior and new features in MySQL 5.7.11. So, read the release notes carefully and prepare to upgrade.

Thursday, February 4, 2016

MySQL Support People - Percona Support

I planned to continue this series of posts with the one about MySQL Support engineers who joined us in Oracle while I was working there, but based on recent events in my life I'd prefer to postpone it and move directly to the team I worked for during last 3+ years, Percona Support Team.

Disclamer: In the list below I still try to pay attention to public contribution by each of engineers mentioned to MySQL Community, mostly in a form of bug reports. This is not the only way they contributed, but I have to stick to this format in current series of posts. I also base my comments mostly on my memory and quick search of public sources of information, so I may be mistaking about the roles, time periods and other details. Please, inform me if you think there is something to fix.

So, here is the list of my former colleagues who formed the best MySQL Support team in the industry in 2013-2015 while working with me in Percona:
  • Miguel Angel Nieto - Miguel was my mentor during my first days in Percona, so I have to start with him. He joined Percona in 2011 and was a team manager of EMEA Support team for a long time. We always managed to work together in a productive and efficient way. Miguel is a real expert in Percona XtraDB Cluster and almost every new technology we had to support, up to MongoDB recently. He used to act as a great consultant as well when Support engineers worked together with Consulting on the same pool of issues, and he still takes SSC (Support Shift Captain) shifts, so he can be the first person from Percona whom you deal with in case of problems even today. I see 13 public bug reports from Miguel and suggest to check his Bug #77654. See also his 12 bug reports for Percona software at
  • Ovais Tariq - when I joined Percona Ovais was the only Principal Support Engineer in the team. He was really great in everything technical he did, from complex data recovery consulting cases to query optimization, and on top of that he was a really good writer. It was a pleasure to read every email in issues he worked on, it was like reading a good technical blog post if not a book. I see 16 public bug reports for MySQL by Ovais. Check his replication-related Bug #70923, for example. He had also reported 20 bugs for Percona software at In 2013 Ovais moved to Consulting and a year later, in 2014, he quit from Percona to become a Lead MySQL DBA - Automation Engineer and later a Lead Reliability Engineer at Lithium. He is also active at TwinDB site.
  • Fernando Ipar - As far as I know, he was the first full time Support engineer and dedicated Support Shift Captain in Percona since September, 2009. For two years later he was a Director of Global Support. When I joined he was working in Consulting and recently he seems to do Development. But during all this time Fernando was interested in Support-related discussions and helped my colleagues promptly when needed. I see 11 bugs reported by him for MySQL, including funny optimizer regression since 5.1.x, Bug #66825. Fernando also reported 10 bugs for Percona software at
  • Michael Rikmas (a.k.a. Mixa) - he moved to Support from Administrative team in 2011, and there he was one of the first Percona employees. He always cared about Percona operations and customers, covering endless SSC shifts any time a day, sometimes for 12+ hours in a row. He was always ready to step into anything any time a day to help, if needed. While working in Support he improved his technical skills a lot in all MySQL-related areas, so he ended up as a good problem solver and eventually quit from Percona (in 2014) to join PSCE as a Consultant. I see only one bug he reported for MySQL, Bug #62426. See also
  • Martin Arrieta - Martin joined Percona in January, 2012, but was already a well known Support engineer when I joined. In 2013 he moved to Consulting and two years later quit from Percona in September, 2015, to join Pythian as Database Consultant. Martin had reported 6 MySQL bugs in public (he was really interested in MySQL Fabric), but he is better known as a blog posts author. I think we always cooperated really well with him. 
  • Marcos Albe (a.k.a. Markus) - he is the most hard working and successful Support engineer of all times in Percona. Marcos was also my team manager after I switched to AMER Support Team. Marcos joined Percona in 2010 and by 2012 he was not only a hard working SSC, but also a full stack expert. Marcos created many useful hints for the internal knowledge base and is also know as a great speaker at Percona Live conferences (I've met him for the first time at Percona Live New York, in 2012, where he had talks on several topics). The way he works (he is ready to call you using any media, join any chat, log in promptly and spend entire night helping any customer interactively) defines famous "Percona style" of providing Support (whatever I may think about it) and gave him huge experience with all technologies even remotely related to MySQL. Marcos had reported 4 public MySQL bugs, including a 5.6.x regression one, Bug #71616, that is still "Verified". I also see 5 bugs reported by him at
  • Nickolay Ihalainen - Nickolay is a classical Percona Consultant (who still works in Support for a long time), with huge real life experience of managing MySQL at scale and full application stack troubleshooting. He knows a lot about Linux, hardware used with MySQL and, as any other great consultant in Percona, is always ready to step into the issue of any complexity and length, call customer, write code, recover InnoDB data or fix a bug quickly. He works in Percona since January 2010, reported 3 MySQL bugs in public, including Bug #78051 that is still "Verified". I also see 7 bugs reported by Nickolay for Percona software at I studied a lot while working with and talking to him.
  • Przemyslaw Malkowski - he joined Percona in August, 2012, just before me. We were at the same "bootcamp" session in famous "Hotel Percona" and worked closely together since that time. Przemek is a Principal Support Engineer since July, 2015. He is the only one who was promoted to that level while I was working in Percona, not just joined like that. He works hard on all kinds of complex issues and is one of key experts in the industry in Galera and Percona XtraDB Cluster (He can easily parse and understand obscure, huge logs from multiple nodes, and explain the interactions and problems  that happened, quickly. This is the skill I always missed). Moreover, he is a well known bug reporter, with 22 bugs reported for upstream MySQL, including infamous Bug #78777 that mislead numerous MySQL users and customers for months, and 37 (!) bugs reported for Percona software at Przemek is able to work with any kind of annoying customer, on issues of any complexity, and still remain helpful. I really enjoyed working with him in one team, a dream team actually!
  • Nilnandan Joshi (a.k.a. Nil) - he joined Percona in May, 2012 and successfully worked there as Support Engineer till January, 2016. He managed to master all key tools and technologies in MySQL, while working hard on boring duties like SSC and bugs processing. Now Nil is a Big Data Engineer at zData Inc and is probably working with Hadoop or something of that kind. He also was visiting "Hotel Percona" with me in 2012 for a "bootcamp", where we spent a lot of time talking about the way InnoDB works (and drinking some beer). Since that time we were working in a close cooperation till his last day in Percona. Nil helped me a lot with bugs processing when I've got a task to manage it from Support side, and was one of few engineers in Support who were ready to report or properly process/verify any bug, build any Percona software, test it on all kinds of weird Linux versions and distros, using all kinds of virtual and real machines. Nil reported 13 public MySQL bugs, including Bug #79469. I also see 9 bugs reported by him for Percona software at I miss our useful work together on bugs and Percona Live sessions already...
  • Jervin Real - he worked in Percona Support since early 2010 and until summer 2013, when he moved to Consulting (he worked great there as well, always ready to step in when we needed help in Support). When I joined he was a key Support engineer for APAC customers. Jervin is a well known blogger, who writes a lot about HA and other technologies for MySQL. I finally met him in real life few days ago, at FOSDEM conference, where he was speaking about TokuDB. Now Jervin is a Technical Service Manager in Percona. He had contributed a lot to MySQL Community also with his 20 public bug reports, including Bug #77715 that is still "Open". I also see 34 bugs reported by Jervin for Percona software at
  • Jaime Sicam - Jaime joined Percona Support in June, 2011 and later became a good manager of APAC team in Support. He always worked hard and helped colleagues with anything, from covering SSC shift to complex troubleshooting. He is a great engineer capable to deal with any kind of issues, but he was our key expert in anything related to PAM and authentication in general. Jaime reported 7 bugs for MySQL software, including infamous Bug #77344, and 11 bugs for Percona software at We rarely were both online and working together, but I already miss him.
  • Muhammad Irfan - he joined Percona Support soon after me, in December 2012. Muhammad actively started to work and cooperate with me and other senior colleagues from the beginning, and soon had become a high skilled engineer in everything related to MySQL, including XtraDB Cluster and bugs processing. He is a quite famous blogger, who was happily writing about new features, tools and best practices (while I prefer to write about bugs and problems, that is, worst practices). I am proud to get a change to help him with some posts as a reviewer. Muhammad spent fair amount of time on bugs processing and reporting as well, as we can conclude from his 8 MySQL bug reports including "Verified" Bug #73094 and 9 bugs reported for Percona software at
  • Fernando Laudares (a.k.a. Nando) - he joined Percona in January, 2013 and worked in AMER Support Team. I'd call him a student of Marcos, as he quickly worked out similar habits and approaches to Support, "Percona style". Fernando always tried his best to help both customers and colleagues, without any limits, and this quickly made him one of the key team members. He works hard, on many issues, day after day without becoming tired in a visible way. Fernando had written many great blog posts and is great in all kinds of automation/testing environments setups. He was fast in studying, accepting and following new technologies we were supposed to support in Percona. His community contributions also include recent enough Bug #77684 and 2 bugs reported at
  • Jericho Rivera - with mostly sysadmin and developer background, Jericho had to work really hard since October, 2013 to reach the level of MySQL expertise we required in Percona Support. But he did that and became a key support provider in 2015, that can deal efficiently with all kinds of customer issues. He helped us to manage Support servers all this time and became a local expert in Docker and other technologies (as you can figure out from his blog posts). He had reported Bug #77073 for MySQL server (still "Verified") and 3 bugs for Percona software at It was really great to see how he becomes better every month, until he ended up just doing an awesome job one day.
  • Peiran Song - I was really happy when she joined us in 2014. By May 2014, just in a couple of months, she already did a great job in Support, working on many complex issues. She was really good with troubleshooting, InnoDB and query optimization. We discussed not only technical details, but also procedures and values of Support, a lot. Peiran reported 6 upstream MySQL bugs, including incredible Bug #73369 that is still waiting for explanation and fix, and 8 bugs for Percona software at It was really sad to meet her in 2015 during Percona Live just to find out that she can not stay with us any more (and not because of any her faults). Peiran is a Data Architect at Smule, Inc now.
  • Justin Swanhart - Justin worked for Percona at different positions (mostly as a Consultant and Instructor), since 2010. In August 2013 he joined Support as a Principal Support Engineer, and since that time we often worked together, in the same AMER team. He had got the Community Contributor of the year 2015 award, for his great contributions including (but not limited to) his software. Justin created 40 MySQL community bug reports, including Bug #76210, and I see 12 bugs reported by him for Percona software at Justin worked well, if you ask me, and helped customers a lot, while being a source of unique expertise and great insights for colleagues and customers (based on his trainer and software developer experience, among other things), but one sad day in June 2015 he found himself fired in the middle of the shift. I tried to find out what was really that much wrong with his work, and found literally nothing myself, so I let others to comment on what happened and why. I think firing Justin was a very bad mistake, even though he had better NOT to comment on it in public in the way he did... I hope we'll work together one day again, maybe even soon.
  • Bill Karwin - Bill worked for Percona since 2010, mostly as a Consultant and Trainer. We were really lucky that he joined Support in 2014 full time (as a Principal Support Engineer and Senior Knowledge Manager). Using his huge experience in SQL and everything MySQL, he started to play a key role in AMER team, working a lot and providing great service to customers. He is very famous for his answers at Quora and StackOverflow. He is a great writer and speaker on MySQL topics, and his book on SQL anti-patters is well known. Bill is literally second to nobody in helping MySQL Community. Check also his 9 MySQL bugs including Bug #73283 (still "Verified"), and 2 bugs at At the end of 2014 Bill left us, probably because it was boring for him to resolve the same problems again and again for customers, instead of doing things right from the very beginning. He is now a Senior Database Architect and Software Engineer at SchoolMessenger.
  • Agustin Gallego (a.k.a. Guli) - I met him first in 2012, at Percona Live New York. He worked for Percona since February, 2012, but joined Support later, in December, 2013. Since that time Agustin worked as SSC and Support Engineer in AMER Support team and played an important role in making it operational and efficient. He had reported Bug #77186 for MySQL and 6 bugs for Percona software at He is gaining experience every day, cooperates well with colleagues and now is a very reliable, respected and useful team member.
  • Roben Paul Namuag - Paul joined Percona Support in July, 2013. In January, 2015 he left us to continue working in Percona as Remote DBA, but we still discuss technical details and, sometimes, life after that. Paul had good and harder times in Support, but he always tried to do his best and played an important role in APAC team, both as SSC and as an experienced engineer working on complex enough issues, even when there was nobody around to help. He had reported 3 MySQL bugs, including Bug #77180 that is till "Open", and 2 bugs for Percona software at I'll remember him as a kind, helpful and friendly person.
  • Akshay Suryavanshi - we chatted with with Akshay a lot before he joined Percona, he was interested in a lot of in-depth details on how MySQL works. Eventually he joined Percona Support, in March 2013, but immediately switched to Remote DBA role and had a great success there, moving later to management positions, up to DBA Team Lead and Senior position in Technical Operations recently. Then, in December 2015, he quit from Percona, to become a MySQL Engineer at Tumblr. He helped community with his blog posts and webinars, and also reported Bug #70818
  • Pablo Padua - he joined Percona in 2014 as a Jr. Support Engineer, and made a lot of progress since that. Unfortunately I can not find any public MySQL bug reports by him, but he worked a lot as SSC in AMER Support Team, and he gets more experience every day. We often worked together on technical issues, when I just provided some background help, checks and comments, while he did real time work for customer. It had been a pleasure to cooperate that way, and, as a result, for many areas in MySQL Pablo now is ready to work without any assistance.
  • Abdelhak Errami - he joined us as Senior Support Engineer in April 2015, when Tokutek was acquired. Since that time he played a key role in TokuDB Support, but also often helped with pure MySQL issues as well. We spent a good time together during Percona Live. Abdel often works on TokuDB bugs, check DB-922 as one of recent examples.
  • Joe Laflamme - I think he also joined us some time in 2015 in the process of Tokutek acquisition. Check DB-809 for one of his TokuDB related bug reports. He coordinated TokuDB bug fixing and support during the period when I describe. Now he is a manager at Percona Technical Services.
I had not mentioned Sveta Smirnova, who joined Percona in March 2015, because she was already mentioned in the previous post. I still plan to write about managers and coordinators of Support who had never reported MySQL bugs in public separately, so stay tuned Tom Basil, Peter Farkas and Erzsebet Olsovzsky. I remember about you and I value your contribution a lot, but engineers first...

I had not named colleagues in Percona who joined after November 1, 2015 explicitly here, and this is the date of the beginning of the entirely new epoch in the history of Percona that should be described separately, if ever.

Sunday, January 17, 2016

MySQL Support People - The Next Generation

My first post in this series caused quite an active discussion on Facebook. Readers correctly noted some mistakes in dates and missing names in the list. I've corrected some of the mistakes already and will correct some others later. What was also noted is that initially support for MySQL was provided by developers (this is really important, we'll get back to this later), and of them many had never even got a title of Support engineer. Of those who had not I listed only Monty...

I just want to explain why I made these mistakes and/or why I (intentionally) had not listed developers etc. First of all, I based my post on my memories (so I could be also wrong about start dates and many details) and that's why I could not list those whom I had never closely worked with. I worked with Monty while in Support and he was involved, to some extent, into my work on the problems that were interesting for me, and he provided support well before I joined. That's why he is in the list. My posts are not about the history of MySQL (or even the history of MySQL Support) - they are about MySQL Support as I was introduced to it in 2005 and about people who made it look and feel as I understand it now - the best, most efficient and useful service for MySQL users.

There are pure managers among these people, those who had never reported any MySQL bug in public probably and rarely (if at all) resolved any pure technical problems for customers. They well deserve separate posts, but here I want to name two of them who played the most important role from my point of few: Tom Basil (Director of MySQL Support in MySQL and Sun till 2008) and Peter Farkas (Director of Support in Percona till 2015).

Now it's time to get back to the topic of this post. "The Next Generation" are engineers who started to provide support for MySQL some time in 2005 or later, when the system as I remember it (created mostly by those named in the first post led by Tom Basil) was already in place, with Eventum as an issue tracking system, public bugs database managed by Support, SOP in place in Wiki, SSC duties defined, and so on. I probably have to include myself into this new generation as well, but this series is not about me.

So, these great engineers joined MySQL Support in MySQL and/or Sun (that had not changed that much in the way we worked, mostly accepted and appreciated our experience) after me or before me, but whom I consider the "next generation" for sure. There are really famous people in MySQL world among them:
  • Geert Vanderkelen - he actually joined in April, 2005, so before me, as Senior Support engineer and MySQL Cluster expert. I just do not understand why I missed him in the previous list and no one corrected me. His list of public bugs reported (I see 171!) is quite impressive. He keeps reporting bugs in public, check Bug #79621, the last one he recently reported. Some day in 2011 he got tires of Support (this happens to many creative people) and moved to development where he works on his Connector/Python project, now in Oracle.
  • Mark Leith - he had became an essential team member immediately when joined, some day in 2005 after me. With this Oracle database background he helped to make the entire MySQL more aware about Oracle database, related tools and approaches (before him there was only Peter Zaitsev who created many worklogs obviously inspired by Oracle, and me, "Oracle agent in MySQL", the rest of the company mostly pretended Oracle never existed, non-standard or they do not care about it, at least until Oracle acquired InnoDB at the end of 2005...). He was a manager of AMER Support team in MySQL and Sun. Mark had reported a lot of bugs in public (I see 177) both while in Support and later when he led MySQL Enterprise Monitor development. Check one of his recent bug reports, Bug #76049. For me he is also a "godfather" of Performance_Schema and father of sys schema in MySQL. Mark is a Senior Software Development Manager in Oracle now and still a very active community member contributing a lot with his blog and presentations.
  • Domas Mituzas - he is all times the greatest MySQL Support engineer I had ever worked with. I probably had to stop at this phrase, but as one of my main job duties used to be "PR manager for Domas" once, I'll continue. He managed to assign 14 issues per single 8 hours shift back in 2008, and resolve 7 or 8 of them completely by the end of that shift. During his first days in MySQL Support in 2005 he managed to teach customer from China how to use Chinese properly in Oracle RDBMS, while doing MySQL support. He managed MySQL in Wikipedia and had his own fork of MySQL maintained before most of current forks appeared. He reported some 53 or so MySQL bugs in public (check also his older account at bugs database), my all times favorite being Bug #12113, that allowed me to resolve at least a dozen of customer issues over years and was fixed somehow only in MariaDB 10.1 recently. He is well known as a speaker about MySQL and cool blogger. Check this his famous post about query cache! His work probably requires a book to describe it, not a post, and he is mentioned in the books. Domas, who joined MySQL Support in October, 2005 is a small data engineer in Facebook since 2009. I am proud that I worked in one team with him, honestly.
  • Chris Calender - he joined probably soon after me and he still provides support in MariaDB. He is an active blogger and bug reporter (I see 69 reports). Check his relatively recent Bug #71395. His posts on building MySQL and MariaDB from source on Windows helped me a lot back in 2012. I always had a huge respect to everything he did.
  • Morgan Tocker - now a MySQL Product Manager in Oracle (this seems to be a popular career path among former MySQL Support engineers...), Morgan joined MySQL Support back in January, 2006. He worked on MySQL in different roles and companies since that, including Percona where he was Director of Training, returning back to Oracle for MySQL Community Manager role. He is an active blogger and bug reporter (I see 41 public bug reports from him). Check his old Bug #28338 that is still "Verified".
  • Shawn Green - he joined MySQL in 2006 and had been a very reliable team member in AMER Support team from the very beginning, great SSC (no wonder, he served in the US Navy) and helpful friend to many of us. He had reported 25 bugs in public as far as I can see. Check his latest still "Verified" public feature request, Bug #74640. He is still a MySQL Senior Principal Technical Support Engineer in Oracle, even though he has enough problems to deal with outside of MySQL during last years.
  • Kyle Joiner - he joined in 2007 based on Shawn's recommendation and with the goal to help us with DB2 Storage engine support (MySQL used to work on IBM System i and use its DB2 database as a storage engine back then). He is probably the strongest MySQL Support engineer, literally. He was a great engineer and SSC in AMER Support (I always felt safe as weekend SSC knowing that Kyle is supposed to be on call, he was always ready to help). Kyle had reported 14 MySQL bugs in public. Check his Bug #48049 that is still "Verified". He quit Oracle to join MariaDB recently, where he is still a Support Engineer.
  • Sean Pringle - he joined MySQL Support in 2006 and worked in APAC Team that I always tried my best in cooperation with. He was a great Support Engineer and also was my guide in San Francisco and Cupertino when I visited USA for the first time in 2007. He left us in 2009 and worked in SkySQL later. I try to keep in touch and still hope to continue working with him one day in one team.
  • Tonči Grgin - he joined MySQL in March, 2006, during last (for me, in MySQL AB life) company meeting in Sorrento. We met in the airport of Munich and later Sinisa, who knew him for decades as a developer and MySQL user, just told me that I have to teach Tonči everything for him to be ready to work in the Bugs Verification Team by the end of the meeting. I did that, and he was my first successful student in MySQL Support. His is my family friend and I had an honor to visit his home near Zagreb and his father's home in Split. Tonči worked on processing bugs for various connectors and complex support issues related to connectors, and later, already in Oracle, he moved to development, to fix those connectors finally, and eventually he became a manager of MySQL Connectors team in Oracle. As far as I know that team recently disappeared and he is now working in Oracle on MySQL performance improvements on Windows. He was always active not only in processing bugs, but also in reporting them, so I see 67 he had reported in public. Check Bug #48228, still "Verified".
  • Sveta Smirnova - she joined probably in May, 2006, but back in Sorrento we were already discussing steps to hire her. As I insisted we should get her into the Bugs Verification Team, I've  got a task to be her mentor, and forced her to work on bugs even before she joined. Long story in short, she managed to follow my steps and do everything I did, but better, in MySQL, Sun, Oracle and now Percona, where she is my colleague and also a Principal Support Engineer. She also do a lot more than I did, as a developer, blogger and presenter at various conferences for years. She is well known as an author of a book on MySQL Troubleshooting published in 2012 by O'Reilly. She is famous. Sveta reported a lot of bugs, check also her older account, 256 in total. Check her latest bug report, Bug #79596. Awful bug, really... To summarize, I am proud to work with her these days, and she is a key member of our team in Percona now.
  • Todd Farmer had reported 327(!) bugs in public and is a very active blogger. He is one of they key contributors to MySQL Community for sure. He joined MySQL as a Support Engineer back in August, 2006 and then was AMER team manger. He was brave enough to step in as a Director of MySQL Support in Oracle back in 2010, and while he is not in my list of top 3 Directors of Support of all times, he still did a lot of useful things for MySQL Support in Oracle. Since 2013 Todd is a Director, Technical Product Management, MySQL in Oracle.
  • Matthey Montgomery - yet another great MySQL Cluster Support engineer and public bug reporter (I see 106 bug reports from him), he joined us in August, 2006. He is a Principal Technical Support Engineer in Oracle now and still deals with NDB. This is incredible! Check Bug #20924, one of his first bug reports that is still "Verified". MySQL had never been able to deal with data types properly, especially in explicit and implicit conversions... Topic for another post, sorry.
  • Johan Idrén - he joined MySQL Support in November, 2006, and worked on all kinds of issues initially, being very productive. Later he specialized in Merlin (a.k.a. MySQL Enterprise Manager now) and when Oracle acquired Sun he decided to move to SkySQL to continue doing MySQL Support. Recently he works as a Systems Engineer in DICE (EA Digital Illusions CE AB). I see 22 public bugs that he had reported, including Bug #28331 that is still "Verified". 
  • Susanne Ebrecht (now Holzgraefe) - yes, we have a family of MySQL support engineers who once worked in the company while not yet a family. Susanne joined us in March, 2007, and is actually of PostgreSQL community origin! She was an active bug reporter, I see 165 bugs she reported, so eventually she ended up in Bugs Verification team where we worked together. She worked mostly on bugs for GUI tools, and one of them is still "Verified", see Bug #55497. Susanne quit with her future husband Hartmut at the end of 2010, when they had either to join Oracle or move elsewhere. She is a consultant now.
  • Andrew Hutchings (a.k.a. Linuxjedi) - I thought he was with us in MySQL AB already, but quick check shows that he joined as MySQL Support Engineer for Sun Microsystems only in 2008. Andrew worked on very complex MySQL Cluster and C API issues. He also developed new features and bug fixes for MySQL and MySQL Cluster in his spare time. Andrew was one of only a few MySQL engineers to win a Sun employee recognition award. Later he had really hard times with Oracle and eventually quit to work on Drizzle. Andrew is a Technical Product Manager at NGINX, Inc now.
  • Gustaf Thorslund - he joined us in October 2007 and was Support Engineer specialized in MySQL Cluster and NDB API. He is probably the best one in anything related to NDB API. With long  breaks he took for parental leave, study etc, he is still working as a Senior (still?) Technical Support Engineer in Oracle. We always had a lot of fun discussing life, work, Ada programming and other matters (excluding NDB that I try to avoid by all means). I see he had reported 17 bugs in public, but they are all closed by now.
  • Gary Pendergast - I am not sure when he joined us, but, as far as I remember, some time after the meeting in Riga (and Sun's acquisition of MySQL) he had quit. Probably it was early 2009 or so. He was a good Support Engineer in APAC Support Team and he is still working with MySQL. Check his blog.
  • Andrii Nikitin - he probably worked in MySQL as developer (of GUI tools) in 2007 already. Later he moved to Support and quickly became a great colleague for us. Switching roles is hard when you are already good in something, but he managed to deal with that as well as with many other hard problems in his life. He actively worked on bugs and I can see 86 bugs he had reported in public. Check his latest public bug that is still "Verified", Bug #77789. He is a Senior Principal Technical Support Engineer in Oracle now and is a core hard working EMEA Support team member.
  • Leandro Morgado - he joined us in Sun in 2008 and worked really well from day one in EMEA team. He used to be active bug reporter and I see 46 his public bug reports, including Bug #43523 that is still "Verified". He is still a Senior(?) Technical Support Engineer in Oracle.
  • Ligaya Turmelle - she joined us in April, 2008 and is well known as a full time Goddess, part time MySQL DBA, occasional PHP programmer and active member of both MySQL and PHP communities as a speaker at various conferences. She probably had several accounts in the public bugs database, but I was able to identify only two of them quickly. Check her Bug #67056 - it's full of words I am trying to avoid by all means, like NDB and memcached... I've got a chance to met her in person at Percona Live 2015 in Santa Clara and we had a drink (or two) together. She is a Principal Technical Support Engineer in Oracle now.
  • Rene' Cannao' - he joined us in Sun in August, 2008, and I remember him as a very capable and reliable Support Engineer in EMEA Team from day one. He had several accounts in public bugs database, so it's hard to aggregate all bugs he had reported over years, but there were many. Check his recent regression Bug #78680. He seems to be working for Dropbox now and also develops his ProxySQL software.
  • Trent Lloyd - he joined MySQL back in June, 2007 and since that time he works in Support (now he is a MySQL Principal Technical Support Engineer in Oracle), often on complex issues including those related to HA, DRBD and Pacemaker and tuning Linux for MySQL. I see 13 his public bug reports, including Bug #43775 that is still "Verified".
  • Mikiya Okuno - he joined MySQL in October, 2007, and was one of those few engineers who ran Japanese language support in MySQL (yes, we did that, with a separate project in Eventum and separate team of engineers in Japan that I worked with a lot!). He joined us from Sun to be soon aquired back by Sun with the rest of MySQL. Mikiya had reported 107 public bug for MySQL software. Check his Bug #65035 that is still "Verified". Mikiya is a MySQL Technical Analyst at Oracle now.
  • Meiji Kimura - he joined MySQL Support in September, 2007 and is a Support Engineer in MySQL KK, Sun and Oracle since that time. He is an active bug reporter and seems to have several accounts in public bugs database. Check his Bug #80018 opened today!
  • Roel Van de Paar leads QA in Percona now. With his 296 public bug reports he is a key contributor to MySQL quality in general. His first public bug report, Bug #41908, is still "Verified" and not fixed. Roel joined MySQL Support (already in Sun) in 2008 and was one of the key support providers in APAC team before he switched to QA. Roel contributes to community (and MySQL Support, indirectly) also with his numerous great posts in Percona's blog.
  • Oli Sennhauser - he was a MySQL Consultant from 2006 or so, but later in Sun he worked in MySQL Support for a couple of years, and I think he was great in this role. He reported 24 bugs recently (for some reason I think he had a second account as well), check the latest of them still "Verified", Bug #78898. Now Oli is a Senior MySQL Consultant and CTO at FromDual GmbH.
  • Bogdan Kecman - he joined us in October, 2007, specialized in MySQL Cluster support and quickly became one of the most useful engineers in this area. His knowledge of hardware also always impressed me. He is still in Oracle and now is probably a member of my former Bugs Verification Team there, as I see his name as bug assignee from time to time. He was an active bug reporter, with 44 bugs reported in public. Check Bug #44760, still "Verified".
  • Ben Krug - he joined back in December, 2007 and played mostly SSC and later TAM role for Facebook. He is a Senior MySQL Support Engineer in Oracle now. Ben was active enough bug reporter, but it's a bit hard to consolidate different accounts he seemed to use over these years to count all bugs. Check his Bug #70056 that I consider really serious for many users and customers who plan to upgrade.
  • Jonathon Coombes - he joined us in Sun in July 2008 as Senior MySQL Support Engineer, mostly to work on MySQL Cluster and MySQL Enterprise Monitor issues. He still works in Oracle and does more or less the same job as Principal MySQL Support Engineer. I remember him as a great team mate. He reported 45 bugs for MySQL software in public, including Bug #48789 that is still "Verified".
  • Roger Nay - he joined us in Sun in October, 2008. He worked a lot on MySQL Enterprise Monitor, as obvious from the list of his 42 public bug reports, including Bug #57574. But he was good in other MySQL software as well. Roger still works as a Senior (?) MySQL Support Engineer in Oracle. Not sure what is his current focus of interests though.
I have to stop for now. The post is already huge... This list is surely not complete (I do remember some other names, for example, but fail to find public bugs they reported) and I am ready to correct it and add new names based on your feedback provided either here in comments or on Facebook.

Stay tuned, next time I'll list those who joined MySQL Support only in Oracle, but still contributed (and still contributing) a lot for MySQL Community.

Sunday, January 10, 2016

MySQL Support People - Those Who Were There First

I'd like to devote this long weekend post, the first in a new series, to my current and former colleagues who once worked or still work in a company that provided public MySQL Support service and had a job role of MySQL Support engineer. The list of companies include MySQL AB/Inc ("good old MySQL"), Sun, Oracle, Percona, MariaDB, FromDual, maybe more (I named only those that surely provided or provides MySQL Support for customers outside of the company).

This is not the first time that I write about "people of MySQL", in a wider sense "those who contribute to MySQL Community". Last time it was about famous MySQL bug reporters. That's because for me, on personal and professional level, MySQL is about people who work on it, not about companies, services or money we all make while working on it. But today I want to concentrate on "MySQL Support People" mostly, and name many of them, those who still do MySQL Support right now when you read this, to those who are now CEOs, CTOs, Directors, VPs, Product Managers, DBAs and Developers. They all have two things in common: they once provided support for MySQL in the same team with me, and their contribution to MySQL Community is visible (can be found easily at or elsewhere). Click on names to see some of their public contributions to MySQL (bugs reported).

You may want to ask why I highlight "Support" in this post? It happened so that recently I have to emphasize in public the importance of Support as a core service for MySQL. Regular readers probably noted this here and there...

Let me write today about those who started to provide MySQL Support in "good old MySQL" before me:
  • Shane Bester - he started to work in MySQL Support back in 2005 and he still actively provides support in Oracle, as far as I know. Click on his name to see a list of 1029 (at the moment of writing) public bug reports he created for MySQL software! Start with Bug #79591 if you are interested in regression bugs in MySQL 5.7. I wish I'd be able to do the magic he does every day with creating repeatable test case for MySQL bugs.
  • Sinisa Milivojevic - rumors say he probably was the first Support Engineer in MySQL. Maybe not, but he does this job to some extent since 1999 or so. I see 74 bugs reported, started from Bug #4. So, he was already contributing to MySQL Community back in 2002 for sure! Recently he seems to verify public bug reports from time to time while still working in Oracle. Not sure what else he does, but does it really matter? He was one of MySQL people who influenced me a lot and I tried to play "better Sinisa" for a couple of years at my current job.
  • Victoria Reznichenko - a well known Queen of MySQL Support in Oracle now probably enjoys her maternity leave (I hope Oracle does not force her to work during this time). She was already a well known Support provider in EMEA when I joined, as one can clearly conclude from the fact that her first Bug #860 (of 102 I can see) was reported back in July 2103, 2 years before I joined MySQL AB. Surely her opinion would always matter for me.
  • Alexander Keremidarski (a.k.a. Salle) - when I joined he was team manager of EMEA Support team. As far as I know he has a similar role now in MariaDB. A lof of MySQL support practices are related to his name in my mind. Based on Bug #11 he had reported, he contributes to MySQL Community at least since 2002. Maybe it was him, not Sinisa, who was the first employee with MySQL Support Engineer title. I let them argue about that.
  • Timothy Smith - he tested my knowledge when they decided to give me Support Engineer job in MySQL. In 2005 he was an APAC team manager. Probably his activity was the reason why I ended up in Support (they were faster). I've used his approach to testing candidates later. I see 130 bugs reported by Tim, and many are still not fixed. I am not sure if he is still related to MySQL in any way, but he built a great team and I always was happy to work with him.
  • Matt Lord - he was already a key Support engineer in AMER team when I joined. 123 public bug reports till he moved to the dark side in Oracle, where he is a MySQL Product Manager now.
  • Harrison Fisk - he probably did consulting or training when I joined, but I clearly remember him in Support soon after that and until he moved to Facebook (after playing TAM role for them from some time). I see 108 bug reports and Bug #53825, for example, makes me wanting to double check current InnoDB code. I wish I have that gdb functions they discuss with Sunny at hand. Harrison had the highest rank in Support all the time we worked together. I would not be surprised if I am still 5+ years behind Harrison in my understanding of InnoDB internals (or anything related to MySQL)...
  • Arjen Lentz - he was a key Support engineer in APAC team when I joined. Now he is the Exec. Director at Open Query Pty Ltd, yet another MySQL services provider. His first public bug report that I can see in the list of 84 total, Bug #108, is almost 13 years old.
  • James Day - he worked all kind of crazy hours (probably he is the only one besides Shane and me who was SSC for all 3 shifts, EMEA, AMER and APAC more than once) and was already a well known guru in complex issues handling back in 2005. I see just 25 public bug reports from him, but he had processed many bugs and many of his comments in InnoDB-related bug reports are priceless. Read Bug #69842, for example. I'll remember our busy night APAC shift we worked together (with nobody else around) forever... He still does MySQL Support in Oracle, as far as I know.
  • Kolbe Kegel - he joined MySQL several weeks before me and he still provides Support in MariaDB. He had always been a very reliable and smart engineer. Check the latest of his 133 public bug reports, Bug #74517.
  • Hartmut Holzgraefe - Hartmut helped me a lot during my very first days in MySQL. He was ready to answer any question, help with code (he probably pushed the patch to have my Celeron CPU supported by the build scripts), whatever I asked. He provided support for NDB cluster, and people who are able to do this form a separate "super team" in MySQL Support that gain my huge respect. The do magic by supporting software I consider NOT supportable without direct communication channel with its developers even today. He had reported 420 bugs as far as I can see, and Bug #77018 is the last "Verified" one of them reported for NDB cluster.
  • Axel Schwenke - he was a key support provider in EMEA when I joined and he is also from that "super team" of NDB-enabled magicians. By the way, originally I was hired to provide NDB support eventually and went to Bugs Verification Team only for some part of usual 6 months probation period. But I skipped getting back to NDB cluster happily, and was able to mostly ignore it for 10 next years... Axel works in MariaDB and recently tried to fight with Oracle on performance benchmarks several times. Check this, Bug #77586, the last of his 69 public bug reports.
  • Miguel Solorzano - my first and only team manager in MySQL/Sun/Oracle. He still manages the same Bugs Verification Team and still works on bugs processing in a very visible way (Google for his name and count hits). His first bug report out of 115 total, Bug #326, is almost 13 years old, and before devoting himself to bugs processing he worked as developer on porting MySQL to Windows. I've got my first email form him with instructions few days before I started, and this email was enough for me to start doing my job from day one. I've forwarded it to maybe a dozen of colleagues who joined later. To summarize, Miguel was an ideal manager for me at some stage. The amount of time and efforts he devoted to MySQL Community is incredible!
  • Bogdan Degtyariov - Bogdan joined MySQL Support few weeks before me and worked on Connectors (mostly ODBC, but others as well, for some time), both when they were mentioned in customer issues and in public bug reports. He had reported 64 bugs. I am not sure what happened to him recently, as his latest public bug report is 5 years old and we had no chance to get in touch since I left Oracle in 2012. I hope he still works on MySQL in Oracle, no matter what is his current job role.
  • Dean Ellis - he was (or soon became) AMER team manger in Support when I joined. He was also a key Support provider for AMER and defined many proper procedures over years. The last of them that still affects every MySQL user was so called "BPS process" for setting priorities of bug fixes in Oracle MySQL. It was him who set up the grounds of the process that was later implemented (from Support side) by Sinisa, me and Sveta Smirnova. I see 140 bugs reported but I am sure he had more than one account and reports date back to 2003. Some of his famous bug reports include Bug #4291 and Bug #1118 (implemented only in 5.7.5+). He was a Vice President, Technical Support at MariaDB last time I checked.
  • Vasily Kyshkin (a.k.a. WAX) - he was a key engineer in my Bugs Verification Team when I joined. No wonder he reported just few bugs - our task was to get the counter of open and not processed bugs smaller, not bigger. It had been <90 for months, not 528 as i see now (and 500+ for months recently). So, engineers from bugs team, mostly reported bugs when requested by customers. I was really sorry when he decided to leave MySQL, but the habit to not care much about Support engineers (and even less - about those working on community bug reports) was not invented recently. Some had it back during "good old MySQL" days...
  • Aleksey Kishkin (a.k.a. Walrus) - according to many, he was the best SSC (Support Shift Captain) of MySQL AB times. I can not argue with that (even if I'd like to think it was me :). He worked a lot and was always ready to help with bugs processing. He left MySQL AB in 2006 probably, then eventually returned. Not sure what is he doing now. As a side note, those engineers with nicknames in this list surely worked since the times were there were just a handful of people in entire MySQL... In 2005 I was asked about the preferred nickname (and I used "openxs", you can search bugs on that, it's still my main UNIX user name everywhere, but it was not widely used in the company).
  • Michael Widenius (a.k.a. Monty) - yes, creator of MySQL provided Support for it and worked in tight cooperation with first Support engineers to define some of the approaches and practices of doing Support. This is what I've heard and can conclude based on his bug reports. Check his Bug #18 to understand the state of MySQL quality back in 2002 :)
  • Jorge del Conde - when I joined Jorge worked in Bugs Verification Team, a lot, on all kinds of bugs. But essentially he was a developer in MySQL since 1998 (one of the very first employees). More than 7 last years he works as Senior Developer at Microsoft. I was really sorry back in 2006 when he had to quit from MySQL Support. Not the last time when I could do nothing to keep a team member, but it's always painful to me.
  • Peter Zaitsev - yes, founder and CEO of Percona was a leader of High Performance Team in Support back in 2005. this team worked on complex performance-related issues and run all kinds of benchmarks. Moreover, Peter had SSC shifts scheduled sometimes, and at least once was late for his shift so I had to call him. He had reported 133 public bugs. Check Bug #59899 as one of the last examples of my communication with Peter from the other side of public bugs database.
  • Vadim Tkachenko - current CTO and co-founder of Percona was the only person from MySQL AB whom I've seen in the real life before I joined. He met me to evaluate if I am good enough to join support. We had some coffee at talked about transaction isolation levels and probably something else. Vadim worked with Peter Zaitsev in High Performance Team and formally was a Performance Engineer, but he worked on usual support issues as well and took SSC shifts. Vadim managed to report 23 public bugs while working in MySQL, and his Bug #14347 is still just "Verified".
  • Alexey Kopytov (a.k.a. Kaamos) - he also worked in High Performance Team in 2005 when I joined, and is famous for his sysbench tool and numerous contributions to Percona Server and Percona XtraBackup since 2010 when he joined Percona. He left Percona somewhat silently some time ago, so I am not sure where exactly he is working know, but he still writes a lot about MySQL and keeps reporting bugs (I see 90 of them) and contributing patches. Check his last bug report and patch in Bug #79487.
  • Lachlan Mulcahy - I remember him as a great APAC team manager after Timothy Smith moved back to USA, but quick check shows that he was a Support Engineer when I joined. List of bugs he reported (49 in total) move us back to September 2004, that's probably when he started. Check his Bug #36151 that is still "Verified". Now he is a Production Engineering Manager at Facebook.
  • Sasha Pachev was a developer in MySQL, but some time in during 2006 or 2007 he worked in Support as well, part time. Check his last public bug report with a patch suggested (out of 8 that I see) that is still "Open", Bug #60593. There are also still "Verified" optimizer bugs reported by him.
  • Jani Tolonen was mostly a developer, but he played many roles and worked in Support part time. His 23 public bug reports covers everything from mysql command line client to Maria storage engine. I am not sure what he is doing now.
  • Indrek Siitan - when I joined and for some time after that he played mostly a "full time SSC" (Support Coordinator) role mostly in Support. But quick check for bugs gives a list of 15, starting with Bug #138, so he surely did other work as well. Now he seems to be a Full-Stack Web Developer (and probably he had always been a web developer in one of his roles).
I studied a lot from all these people while working with them. They surely know what MySQL Support is about and contributed a lot both to MySQL Community and to my personal ideas on what's good and bad in MySQL services.

UPDATE: my readers on Facebook noted that there were other engineers as well who worked in Support or who had provided Support in MySQL before I joined and before many people of those I've listed started to do that. I tried to work based on my memory and also, just because of format of this blog, I had to list those who has several real bugs reported in public. I did some corrections and additions in the list based on feedback I've got.

If I missed your name in the list or had written something about your work that is not correct, please, forgive me my old memory and correct me.

Next time I'll write about those great engineers who joined MySQL Support after me and became much better than I'd ever be able to become. Stay tuned!

For now just remember the following statement that I've used more than once in different discussions already:
"There are two kinds of people: people of MySQL and all other people. You can not judge them by the same rules."
I had not changed my mind since the first time I stated that while defending one of colleagues in internal discussion.

Saturday, January 9, 2016

"I'm Winston Wolf, I solve problems."

My (few) readers are probably somewhat tired of boring topics of metadata locks and gdb breakpoints that I discuss a lot this year, so for this weekend I decided to concentrate on something less technical but still important to me - the way I prefer to follow while providing support for MySQL.

Before I continue, it's time to add the explicit disclaimer: the views on how support engineer should work expressed below are mine alone and not those of my current (or any previous) employer. Specific case I describe may be entirely fictional and has nothing to do with any real life customer. I love thy customers in reality...

One of my favorite movies of all times is Pulp Fiction. Coincidentally, it was released in 1994, more or less at the same time when providing technical support started to become one of my regular job duties, not just a hobby or boring part of sysadmin job role I had to play even when not wanted. I had to provide support for the software I had written as soon as it had got first customer, then moved on to helping my colleagues (whom we developed software with together) with all kinds of technical problems they had, from proper coding style to linking Pro*C programs to, well, getting more disk space on NFS server that was actually my workstation. In 10 years or so this ended up with sending CV to MySQL AB and getting a Support Engineer job there (instead of a Developer one I applied for).

So, with this "support" role that I played already when I watched the "Pulp Fiction" for the first time, it was natural for me to find out that my favorite character there is "The Wolf". As soon as I started to provide MySQL Support, I've added the quote that is used as a title of this post to my LJ blog, as a motto. This person and his approach to "customers" of his very specific "service" (resolving all kinds of weird problems) looked ideal to me, and over years this had not changed. Let me remind you this great dialog with Vincent Vega (see the script for the context and details if you do not remember them by heart; I've fixed one typo there while quoting, bugs are everywhere, you know...):

               The Wolf and Jimmie turn, heading for the bedroom, leaving 
               Vincent and Jules standing in the kitchen.

                              (calling after him)
                         A "please" would be nice.

               The Wolf stops and turns around.

                                     THE WOLF
                         Come again?

                         I said a "please" would be nice.

               The Wolf takes a step toward him.

                                     THE WOLF
                         Set is straight, Buster. I'm not 
                         here to say "please."I'm here to 
                         tell you what to do. And if self-
                         preservation is an instinct you 
                         possess, you better fuckin' do it 
                         and do it quick. I'm here to help.
                         If my help's not appreciated, lotsa 
                         luck gentlemen.

                         It ain't that way, Mr. Wolf. Your 
                         help is definitely appreciated.

                         I don't mean any disrespect. I just 
                         don't like people barkin' orders at 

                                     THE WOLF
                         If I'm curt with you, it's because 
                         time is a factor. I think fast, I 
                         talk fast, and I need you guys to 
                         act fast if you want to get out of 
                         this. So pretty please, with sugar 
                         on top, clean the fuckin' car.

Over the years of doing support I've found out how important it is to tell it straight and honestly from the very beginning: "I'm not here to say "please."I'm here to tell you what to do."

In services the approach is often the following: "The customer is always right". In reality, speaking about customers of technical services at least, it may NOT be so. Customer may be wrong with his ideas about the root cause of the problem, and (as the wiki page linked above says) can be dishonest, have unrealistic expectations, and/or try to misuse a software. What's even more important, customers are rarely right when they say how services should be provided for them.

All these:
"Can I speak on Skype with Valerii? ... Can I chat with him on Skype please? It's a lot easier ... and I have questions" 
may end up with a chat, or may end up with an email that was already sent while this chat with oncall engineer happened, explaining both possible root causes of the problem, asking followup questions to define the real root case, and listing next steps to pinpoint or workaround the problem.

As this is still a technical blog, the problem in the case I had in mind while writing the above, was a very slow query execution on MariaDB 10.x, where query had 80K IDs in the IN list and previously it was executed fast on Percona XtraDB Cluster 5.5.x. The range of root causes to suspect was wide enough initially, from Bug #20932 to Bug #76030 and maybe to some MariaDB specific bugs to search for, to disk I/O problems, lock waits (what if the SELECT query was executed from SERIALIZABLE transaction). So, I kept asking for more diagnostic outputs and insisting on getting them in emails, and yes, long outputs are better shared via emails! Sharing them in chat or, even more, discussing over phone, or seeing them over a shared desktop session (the approaches many customers insist on) is neither faster nor more convenient. I say what to execute, you do that, copy/paste the output and send email (or reply in the issue via web interface, if you prefer to paste there), "pretty please, with sugar on top".

As soon as evidence provided shown there is no locking or disk I/O problem, my investigation was concentrated around this important fact: it was fast on 5.5.x and is slow on MariaDB 10.x with the same data. What is the difference in these versions that matter? Most likely it's in the optimizer and new optimizations they have in MariaDB! 

Do I know all the MariaDB optimizations by heart? No, surely, I have to go read about them, check how proper optimizer switches are named and make up my mind about suggestions. Is it OK to do all these while hanging on phone with customer or chatting with him? Well, maybe, if customer prefers to listen to my loud typing and sounds in my neighborhood... I'd prefer NOT to listen to any of that, ever, and not to hang on the chat with 10 minutes in between messages. So, the only chat reply customer got was: "Please reply to his request and I'll have him follow up."

The last my request at this moment was simple:
"I do not see optimizer_switch set explicitly in your my.cnf, so I assume defaults there as shows. Can you, please, check if setting this:

set optimizer_switch='extended_keys=off';

before running the problematic query allows it to run faster?
Even though the bug that led me to this idea (after reading the details of what optimizations MariaDB provided and how they are controlled) is still "Open", and I had never been able to create a test case not depending on any customer data, I've seen the problem more than once. I think fast, and I type fast, so I shared this suggestion immediately (1h 20 minutes after customer started to describe the problem in chat, 43 minutes after customer provided all the outputs required in email). And you know what happened? Chat continued more or less like this:
"wow... optimizer_switch with extended_keys off worked... from 1000k+ seconds to ... 1 second ... rofl ...  so extended_keys=on by default ... it literally halted our database today almost ... thank you for your assistance ... i would have never of thought to change that... and you guys caught it, so props"
So, the immediate problem was resolved and, you know, the resolution started with my explicit refusal to join any chat until I see the evidence and outputs requested in emails. It also ended with email suggesting to switch off one specific optimization that is used by default and is known to me, the one pretending to be expert, from several previous cases (similar or not so much, as I never seen anything like this this happening on MariaDB before). It took 1 hour 20 minutes from initial problem statement communicated to problem resolved, and all this time I worked asynchronously and concurrently on this and few other issues, and had not said or typed a word to customer in chat.

This is how The Wolf solves problems: 
"I'm here to tell you what to do. And if self-preservation is an instinct you possess, you better fuckin' do it and do it quick. I'm here to help. If my help's not appreciated, lotsa luck gentlemen"
I do it the same way, and. IMHO, any expert should do it this way. If a customer always knows better what to do and how to communicate, why they ended up with a problem that brought them to me? 

I try to prove every point I make while working on problems, and I expect from the other side to apply the same level of efforts - they have to prove me the way they want issue handled is better than the one I prefer and suggest, if they think this is the problem. After that I'll surely do what's the best from them, knowing why do we both do it so. 

In reality, the problem usually lies elsewhere... Now, it's time to re-read my New Year wishes post.