There are three solutions for providing consistency in a data service that operates across a wide area network (WAN). None of them are free. What are you willing to pay and where are you willing to add complexity? Depending on what you choose your system can be more complex for external users, internal application developers or operations. The choices are multi-master with conflict resolution (eventual consistency), multi-master with conflict prevention (strong consistency) and single master with downtime on failover.
If you choose eventual consistency (EC) then internal application developers must write logic to resolve conflicts and external users will occasionally encounter inconsistent data. This might be a small price to pay for a system that provides higher availability and transparent failover. I am not aware of support for secondary indexes in the popular EC systems. I wonder if the same logic that does eventual consistency across a WAN might be reused to keep secondary indexes eventually consistent within a datacenter. That would impose an additional cost on internal application developers in return for expanding the workloads that EC can support.
If you choose strong consistency then external users experience more latency on writes as the transaction commit requires one or two round trips across a WAN. This might be a small price to pay for a system that provides higher availability and transparent failover. Galera is doing interesting work in this area for MySQL and they have already begun to publish results. I need to read more about that.
If you choose single master then you will spend more money to make that master less likely to fail. You will also experience more downtime and higher support costs while doing manual failover as quickly as possible. Solutions include RAID 10, battery backed write cache, highly-available SAN/NFS, DRBD and pagers for your operations team.
I don't know if people choose single master in the MySQL community. There are not many choices. It supports multi-master replication but without conflict resolution. It supports strong consistency with Galera but that is new on the MySQL market. Galera might be the killer application for MariaDB. Tungsten is another product that can reduce the complexity of master-slave replication.
Unless you are using Tungsten, it very hard to automate master failover for MySQL when there is more than one slave per master. But many deployments need a master and slave in one datacenter and another slave in a remote datacenter.
Sunday, April 25, 2010
Subscribe to:
Post Comments (Atom)
Posts
FYI: MySQL Cluster geographical replication supports conflict detection/resolution. The current support is quite rudimentary, but we're improving it (and it do support secondary indexes)
ReplyDeleteThanks for the mention of Tungsten. Your take on automation of master failover is exactly the motivation behind our management--you either need to be automatic and fast on master/slave (Tungsten) or go to true multi-master. We also are working on a feature to allow users to choose to master shards in different locations, which is sort of a half-way stop between master/slave and full multi-master.
ReplyDeleteAlexey Yurchenko's recent post on Galera WAN replication is thought provoking. I think the interesting question for Galera is perhaps not how well virtual synchrony works when you get a partition but how well it works when things get congested over the net, which is quite common. At that point you might be out to your 99% case for anything participating in distributed transactions. I'm looking forward to more posts from Alexey on this.
I need a standard disclaimer for all of my posts. I don't understand Cluster yet and it frequently does a lot better for some of the behavior I am trying to improve.
ReplyDeleteI am also not current on Tungsten and Galera as I will spend most of this year upgrading to the most recent version of MySQL.
@Jonas Oreland
ReplyDeleteDo you have a URL for docs on geo distributed conflict resolution for cluster?
-- Rob Wultsch
@Rob http://dev.mysql.com/doc/refman/5.1/en/mysql-cluster-replication-conflict-resolution.html
ReplyDelete@Rob http://dev.mysql.com/doc/refman/5.1/en/mysql-cluster-replication-conflict-resolution.html
ReplyDeleteBefore going the expensive WAN route, I’d like to view the consistency solution in terms of High Availability (HA) needs.
ReplyDeleteThere are two types of situations that require HA. The first is when you want to maintain HA against hardware or human error (such as accidentally deleting data). For these situations, simply replicating among a few local MySQL instances, serves the purpose. No need to invest in a WAN solution.
The second is when you want to maintain HA against large local area disasters, such as earthquakes and power outages. Only in these situations do you need to replicate across a WAN.
It’s reasonable to expect some downtime (in terms of minutes) during failover, during which you perform a database failover and other services to a second location. Clustering across the WAN is not desirable due to high latency. That’s why few people do it. MySQL replication is normally good enough for most of these cases.
The key to minimizing data loss across WAN replication is to keep each transaction as small as possible. In MySQL, only after a transaction is committed on the master, is the transaction shipped over to the slave. The sooner each transaction is shipped to the slave, the less data is lost if the master is dead.