While debugging a problem, I read some of the code used to send binlog events from the master to a slave. I wanted to determine when data for replication is written to the socket by the master. The protocol for this is interesting. Data for binlog events are written to a buffer on the master as soon as the events are available. The code for this is in
mysql_binlog_send which is run by the thread on the master associated with one slave. The data is sent to the slave (written to a socket) in two cases:
- The buffer is full. I think the size of the buffer is determined by the value of the net_buffer_length variable. This is set to 16384 on my servers.
- There are no more binlog events to send.
An implication of this is that when a master is busy and the IO thread on a slave remains behind the master, it will receive replication events in
net_buffer_length byte chunks and some replication events will remain in a buffer on the master until the buffer is full, rather than be sent as soon as they are available.
I don't know how much this contributes to replication delay. When semi-sync replication as implemented in the
Google patch is used, the buffer is usually flushed at transaction boundaries whether or not the buffer is full. This is done so that a slave can acknowledge receipt of the binlog events before the transaction commit returns to a user.
Whether or not semi-sync replication is deployed, flushing data to the slave on transaction boundaries decreases the chance of transaction loss on failover.
Posts
When tuning synchronization delay, is it possible to specify a time parameter, opposed to a buffer size? For instance, what about sending binlog data asynchronously every 5 seconds, regardless if the buffer is full.
ReplyDeleteWe are exploring a master-master scenario and researching ways to minimize this lag, if possible. And hopefully going full-tilt to MySql Clusters. =)
To minimize replication delay first make sure that the hardware for slave servers is similar to that for masters. The next thing to do is to avoid long-running statements on the master that must get replayed by the slave.
ReplyDelete