I spent more time looking at the performance regressions in MySQL 5.6 for single-threaded read-only workloads and I can get 5.6 closer to 5.1 by disabling a few options when building MySQL 5.6 -- optimizer tracing & profiling. This is my summary of things to do if you want to avoid some of the performance regressions I encountered while doing update-only and read-only performance tests for single-threaded and high-concurrency workloads:

• Disable optimizer tracing --  use -DOPTIMIZER_TRACE=0 with cmake
• Disable profiling -- use -DENABLED_PROFILING=0 with cmake
• Disable performance schema -- use -DWITH_PERFSCHEMA_STORAGE_ENGINE=0 with cmake. See bug 68413 and many blog posts from me.
• Set metadata_locks_hash_instances=256 -- see bug 68487
• Set innodb_checksum_algorithm=CRC32 -- but read the manual first! This isn't a new problem. This is a faster checksum algorithm not available in older versions of InnoDB.
• Get the fix for bug 68588 to avoid stalls on sleep during fsync. This isn't a new problem. There is more work in progress to fix new regressions in InnoDB. This is my placeholder for the problem in general.
• Set innodb_io_capacity and innodb_lru_scan_depth to values that are large enough given the IO capacity of your server. This wasn't as important in the past when InnoDB did furious flushing. This is very important in 5.6. More details are here.

I repeated tests for single-thread read-only sysbench after disabling optimizer tracing and profiling as described above and this reduced the 5.6 regression. I think the remaining overhead is in the optimizer.

• 11916 QPS -- orig5163
• 11795 QPS -- fb5163
• 11286 QPS -- orig5610, no perf schema, optimizer tracing, profiling
• 10875 QPS -- orig5610, no perf schema

This post describes MySQL 5.6 performance for a workload that is update-only and single-threaded. My previous post was for a read-only workload. I think there is a performance regression in MySQL 5.6 for a workload dominated by fast queries even when the performance schema is disabled. I filed bug 68825 for this. Hopefully my next post will identify the sources of that regression.

These tests used sysbench with 128M rows in one table. I compared MySQL 5.6.10 (orig5610 below), MySQL 5.1.63 (orig5163 below) and MySQL 5.1.63 with the Facebook patch (fb5163 below). For all tests the sysbench clients ran on the same host as mysqld. The sysbench workload is to update one row by primary key per query. The database table is about 29GB on disk. The binlog and InnoDB doublewrite buffer were disabled during the test. Fsync on commit was also disabled (innodb_flush_log_at_trx_commit=2 was used). The test server has fast storage and can do 100 microsecond reads of 16kb pages for a single-threaded read-only workload. Write latency is worse when many random writes are done.

I looked at PMP stack traces during the test and I don't think the problem is InnoDB. My guess is that the problem is similar to what was measured on the read-only tests -- more code was added to 5.6 that makes query optimization or execution slower.

IO-bound

Tests were first run for an IO-bound configuration using a 4G InnoDB buffer. Results for orig5163 are probably worse because of the overhead from computing InnoDB checksums as described in the read-only post. Enabling the perf schema with default options costs about 10% of peak performance for MySQL 5.6. MySQL 5.1 is able to match 5.6 performance when the Facebook patch is used because of the innodb_fast_free_list option that improves the efficiency of finding clean pages from the LRU tail. In the results below iocap means innodb_io_capacity and ffl means innodb_fast_free_list. When not specified the server used 8192 for innodb_io_capacity and innodb_lru_scan_depth.

• 4160 QPS - fb5163, iocap=1024, ffl=1
• 4107 QPS - orig5610, performance_schema not compiled
• 4100 QPS - fb5163, iocap=1024, ffl=0
• 3909 QPS - orig5610, performance_schema=ON, default options
• 3596 QPS - fb5163
• 3352 QPS - orig5163

Cached

Tests were repeated using a 64GB InnoDB buffer pool. The table was cached by InnoDB prior to the test but the adaptive hash index was cold so most UPDATE statements spent some time updating the AHI. Unlike the IO-bound tests, 5.6 was slower than 5.1 and again the perf schema has a significant overhead -- 9.4%.

• 9231 QPS - orig5163
• 9089 QPS - fb5163
• 7810 QPS - orig5610, performance_schema not compiled
• 7365 QPS - orig5610, performance_schema=ON, default options

LinkBench is a benchmark that models the social graph OLTP workload at a busy social networking site. It is implemented for MySQL but there is abstraction layer so it should be possible to implement drivers for other products like HBase, Cassandra, MongoDB and Postgres. This has been a long running project that was finished by an awesome intern. I am happy to share the announcement and source code. I think we will soon have results from Percona for a few MySQL variants and I hope that the benchmark is useful to others.