Fabric结合GTID实现Mysql复制高可用(1)
mysql版本:Percona server 5.6.19-67.0 Percona Server (GPL), Release 67.0, Revision 618
Mysql安装参考:http://my.oschina.net/anthonyyau/blog/284092
Fabric State store安装参考:http://my.oschina.net/anthonyyau/blog/307165
半同步复制参考:http://my.oschina.net/anthonyyau/blog/269800
环境:
4个服务器实例运行Percona server 5.6.19-67.0,fabric node安装在单独的服务器,每个Mysql实例在一台服务器上;
fabric为启用复制的3个mysql实例提供高可用,应用使用fabric-aware的连接器路由事务和SQL语句到合适的服务器,透明的进行读和写操作;
当前仅仅支持异步primary backup复制(半同步复制需要手工完成)。primary处理所有写操作,Secondaries使用mysql复制从primary同步,可以进行读操作。
服务器信息:

一、mysql实例和fabric state store准备
Mysql安装好后,每个fabric管理的实例需要开启gtid、二进制日志(不需要启动复制,由fabric完成,但是不支持配置异步复制),将以下配置放到[mysqld]段下面:
- log-bin=mysql-bin
- binlog_format=ROW
- server-id = 29 #保证每个mysql实例唯一
- log-slave-updates=true
- gtid-mode=on
- enforce-gtid-consistency=true
- sync-master-info=1
数据库用户账号准备:
1、超级管理员账号,本例使用root@'172.17.42.1',密码为admin@123;
2、fabric管理mysql实例的账号,本例使用fabric@'172.17.42.1',密码为fabric@456;
3、mysql复制专用账号,本例使用fabric@'172.17.0.%',密码为fabric@456,需要与fabric管理mysql的账号密码一致;
查看mysqlfabric命令帮助:
mysqlfabric help:显示简短的语法信息和帮忙命令
mysqlfabric help commands: 列出所有可用命令和描述
mysqlfabric help groups: 列出可用命令组
mysqlfabric help [group] [command]: 提供命令的详细帮助信息
二、使用Fabric创建mysql复制高可用组
1、创建组
- # mysqlfabric group create my_group
- Password for admin:
- Procedure :
- { uuid = 3f7e82bc-4291-4002-8688-1929fc63ed3e,
- finished = True,
- success = True,
- return = True,
- activities =
- }
要输入xml-rpc密码,可以将密码指定到fabric的配置文件,或者设置disable_authentication = yes,需要重启fabric(先mysqlfabric manage stop,然后修改配置文件,不然将报"Permission denied."错误)。
2、添加mysql实例到组
- # mysqlfabric group add my_group 172.17.0.50:3306
- Procedure :
- { uuid = 6e69a5a7-667b-4e63-92c3-2f9f4269d633,
- finished = True,
- success = True,
- return = True,
- activities =
- }
- # mysqlfabric group add my_group 172.17.0.47:3306
- Procedure :
- { uuid = 70a66b03-a16d-426d-a711-8f13da78fc8d,
- finished = True,
- success = True,
- return = True,
- activities =
- }
- # mysqlfabric group add my_group 172.17.0.48:3306
- Procedure :
- { uuid = 15a2e1ad-b726-4fa8-bdf3-a1701e870166,
- finished = True,
- success = True,
- return = True,
- activities =
- }
添加实例到组错误:
- # mysqlfabric --param=servers.user=fabric --param=servers.password=fabric@456 group add my_group 172.17.0.50:3306
- Password for admin:
- Procedure :
- { uuid = d2c0d969-5b8a-40c2-ba7a-1bc963c08824,
- finished = True,
- success = False,
- return = ServerError: Error accessing server (172.17.0.50:3306).,
- activities =
- }
日志:
- [DEBUG] 1409151193.274349 - Executor-4 - Statement (BEGIN, Params(()).
- [DEBUG] 1409151193.274764 - Executor-4 - Executing _add_server
- [DEBUG] 1409151193.274849 - Executor-4 - Statement (SELECT group_id, description, master_uuid, master_defined, status FROM groups WHERE group_id = %s, Params(('my_group',)).
- [DEBUG] 1409151193.275494 - Executor-4 - Start executing function: discover_uuid((), {'connection_timeout': 5, 'address': '172.17.0.50:3306'}).
- [DEBUG] 1409151193.276136 - Executor-4 - Error executing function: discover_uuid.
- [DEBUG] 1409151193.276203 - Executor-4 - _add_server failed, executing compensation
- [DEBUG] 1409151193.276255 - Executor-4 - Error accessing server (172.17.0.50:3306).
- [DEBUG] 1409151193.276308 - Executor-4 - Statement (ROLLBACK, Params(()).
- [DEBUG] 1409151193.276664 - Executor-4 - Complete job (60b0fa1f-3a39-43f3-aa71-46ea732aae84, 946e0199-53ca-4fdc-9b49-f134d44db476, mysql.fabric.services.server._add_server, Error).
将密码写到fabric配置文件是可以,使用命令行参数不能覆盖配置文件参数,暂时没有找到是什么原因。
3、查看组中信息
可以看到所有实例的状态都是SECONDARY
- # mysqlfabric group lookup_servers my_group
- Command :
- { success = True
- return = [{'status': 'SECONDARY', 'server_uuid': '19a37552-2d44-11e4-af5c-763d1493518d', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.50:3306'}, {'status': 'SECONDARY', 'server_uuid': '7bd52611-2d44-11e4-af5f-3ecad7c2f82a', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.47:3306'}, {'status': 'SECONDARY', 'server_uuid': 'ade3ee53-2d44-11e4-af60-de532998e8a6', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.48:3306'}]
- activities =
- }
4、查看组健康详细信息
- # mysqlfabric group health my_group
- Command :
- { success = True
- return = {'19a37552-2d44-11e4-af5c-763d1493518d': {'status': 'SECONDARY', 'is_alive': True, 'threads': {'is_configured': False}}, '7bd52611-2d44-11e4-af5f-3ecad7c2f82a': {'status': 'SECONDARY', 'is_alive': True, 'threads': {'is_configured': False}}, 'ade3ee53-2d44-11e4-af60-de532998e8a6': {'status': 'SECONDARY', 'is_alive': True, 'threads': {'is_configured': False}}}
- activities =
- }
5、提升和降级master服务
创建高可用组后,fabric没有意识到任何复制拓扑。需要提升一个为primary,降级剩余的服务器自动为secondaries(slaves)。
查看命令帮助:
# mysqlfabric help group promote
group promote group_id [--slave_id=NONE] [--update_only] [--synchronous]
如果只是想更新state store,跳过复制配置,使用--update_only参数。
如果slave没有提供,将选择一个最好的候选者,候选者必须开启二进制日志,同时跟master属于同一个组,跟master延时小。进行故障切换操作,选择这个候选者,同时将其他slave指向到新的master,同时更新state store。
提升一个mysql实例为master:
- # mysqlfabric group promote my_group
- Procedure :
- { uuid = 21eb4d58-d7ec-41eb-a0e5-560eb5976272,
- finished = True,
- success = True,
- return = True,
- activities =
- }
执行同样的命名将设置不同的服务器为primary,同时降级当前primary并选取一个新的。如果当前primary错误,执行同样的命令能手动触发选取一个新的primary。
一个标记位"faulty"状态的服务器不能提升为secondary或primary,需要先转换成"spare"状态。使用命令mysqlfabric server set_status <server-address> spare
如果直接从"faulty"转换成"secondary"将报错:
- # mysqlfabric server set_status 3ecc746f-2e05-11e4-b448-560d7281695e secondary
- Procedure :
- { uuid = 06d0acb1-ad01-483b-a1d4-e13c4b775fcd,
- finished = True,
- success = False,
- return = ServerError: Cannot change server's (3ecc746f-2e05-11e4-b448-560d7281695e) status from (FAULTY) to (SECONDARY).,
- activities =
- }
查看组状态和验证是否复制正常:
- # mysqlfabric group lookup_servers my_group
- Command :
- { success = True
- return = [{'status': 'PRIMARY', 'server_uuid': '19a37552-2d44-11e4-af5c-763d1493518d', 'mode': 'READ_WRITE', 'weight': 1.0, 'address': '172.17.0.50:3306'}, {'status': 'SECONDARY', 'server_uuid': '7bd52611-2d44-11e4-af5f-3ecad7c2f82a', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.47:3306'}, {'status': 'SECONDARY', 'server_uuid': 'ade3ee53-2d44-11e4-af60-de532998e8a6', 'mode': 'READ_ONLY', 'weight': 1.0, 'address': '172.17.0.48:3306'}]
- activities =
- }
使用show slave status查看slave状态:
- # mysql -ufabric -pfabric@456 -h172.17.0.47 -e "show slave status\G;"
- *************************** 1. row ***************************
- Slave_IO_State: Connecting to master
- Master_Host: 172.17.0.50
- Master_User: fabric
- Master_Port: 3306
- Connect_Retry: 60
- Master_Log_File:
- Read_Master_Log_Pos: 4
- Relay_Log_File: d4f404f647b0-relay-bin.000001
- Relay_Log_Pos: 4
- Relay_Master_Log_File:
- Slave_IO_Running: Connecting
- Slave_SQL_Running: Yes
- Replicate_Do_DB:
- Replicate_Ignore_DB:
- Replicate_Do_Table:
- Replicate_Ignore_Table:
- Replicate_Wild_Do_Table:
- Replicate_Wild_Ignore_Table:
- Last_Errno: 0
- Last_Error:
- Skip_Counter: 0
- Exec_Master_Log_Pos: 0
- Relay_Log_Space: 151
- Until_Condition: None
- Until_Log_File:
- Until_Log_Pos: 0
- Master_SSL_Allowed: No
- Master_SSL_CA_File:
- Master_SSL_CA_Path:
- Master_SSL_Cert:
- Master_SSL_Cipher:
- Master_SSL_Key:
- Seconds_Behind_Master: 0
- Master_SSL_Verify_Server_Cert: No
- Last_IO_Errno: 1045
- Last_IO_Error: error connecting to master 'fabric@172.17.0.50:3306' - retry-time: 60 retries: 31
- Last_SQL_Errno: 0
- Last_SQL_Error:
- Replicate_Ignore_Server_Ids:
- Master_Server_Id: 0
- Master_UUID:
- Master_Info_File: /usr/local/Percona-Server-5.6.19-rel67.0-618.Linux.x86_64/data/master.info
- SQL_Delay: 0
- SQL_Remaining_Delay: NULL
- Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
- Master_Retry_Count: 86400
- Master_Bind:
- Last_IO_Error_Timestamp: 140827 16:42:50
- Last_SQL_Error_Timestamp:
- Master_SSL_Crl:
- Master_SSL_Crlpath:
- Retrieved_Gtid_Set:
- Executed_Gtid_Set: 7bd52611-2d44-11e4-af5f-3ecad7c2f82a:1-6
- Auto_Position: 1
看到连接错误:
因fabric使用其连接管理mysql实例的用户配置slave,所有需要重新授权slave访问master:
- # mysql -uroot -padmin@123 -h172.17.0.50 -e "grant replication slave on *.* to fabric@'172.17.0.%' identified by 'fabric@456'"
- # mysql -uroot -padmin@123 -h172.17.0.47 -e "grant replication slave on *.* to fabric@'172.17.0.%' identified by 'fabric@456'"
- # mysql -uroot -padmin@123 -h172.17.0.48 -e "grant replication slave on *.* to fabric@'172.17.0.%' identified by 'fabric@456'"
授权后重新查看slave状态:看到连接正常,复制正常工作
- # mysql -ufabric -pfabric@456 -h172.17.0.48 -e 'show slave status\G'
- *************************** 1. row ***************************
- Slave_IO_State: Waiting for master to send event
- Master_Host: 172.17.0.50
- Master_User: fabric
- Master_Port: 3306
- Connect_Retry: 60
- Master_Log_File: mysql-bin.000004
- Read_Master_Log_Pos: 3451
- Relay_Log_File: e294ab366580-relay-bin.000003
- Relay_Log_Pos: 448
- Relay_Master_Log_File: mysql-bin.000004
- Slave_IO_Running: Yes
- Slave_SQL_Running: Yes
- Replicate_Do_DB:
- Replicate_Ignore_DB:
- Replicate_Do_Table:
- Replicate_Ignore_Table:
- Replicate_Wild_Do_Table:
- Replicate_Wild_Ignore_Table:
- Last_Errno: 0
- Last_Error:
- Skip_Counter: 0
- Exec_Master_Log_Pos: 3451
- Relay_Log_Space: 4169
- Until_Condition: None
- Until_Log_File:
- Until_Log_Pos: 0
- Master_SSL_Allowed: No
- Master_SSL_CA_File:
- Master_SSL_CA_Path:
- Master_SSL_Cert:
- Master_SSL_Cipher:
- Master_SSL_Key:
- Seconds_Behind_Master: 0
- Master_SSL_Verify_Server_Cert: No
- Last_IO_Errno: 0
- Last_IO_Error:
- Last_SQL_Errno: 0
- Last_SQL_Error:
- Replicate_Ignore_Server_Ids:
- Master_Server_Id: 27
- Master_UUID: 19a37552-2d44-11e4-af5c-763d1493518d
- Master_Info_File: /usr/local/Percona-Server-5.6.19-rel67.0-618.Linux.x86_64/data/master.info
- SQL_Delay: 0
- SQL_Remaining_Delay: NULL
- Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
- Master_Retry_Count: 86400
- Master_Bind:
- Last_IO_Error_Timestamp:
- Last_SQL_Error_Timestamp:
- Master_SSL_Crl:
- Master_SSL_Crlpath:
- Retrieved_Gtid_Set: 19a37552-2d44-11e4-af5c-763d1493518d:1-10
- Executed_Gtid_Set: 19a37552-2d44-11e4-af5c-763d1493518d:1-10
- Auto_Position: 1
- # mysql -ufabric -pfabric@456 -h172.17.0.47 -e 'show slave status\G'
- *************************** 1. row ***************************
- Slave_IO_State: Waiting for master to send event
- Master_Host: 172.17.0.50
- Master_User: fabric
- Master_Port: 3306
- Connect_Retry: 60
- Master_Log_File: mysql-bin.000004
- Read_Master_Log_Pos: 3451
- Relay_Log_File: d4f404f647b0-relay-bin.000004
- Relay_Log_Pos: 448
- Relay_Master_Log_File: mysql-bin.000004
- Slave_IO_Running: Yes
- Slave_SQL_Running: Yes
- Replicate_Do_DB:
- Replicate_Ignore_DB:
- Replicate_Do_Table:
- Replicate_Ignore_Table:
- Replicate_Wild_Do_Table:
- Replicate_Wild_Ignore_Table:
- Last_Errno: 0
- Last_Error:
- Skip_Counter: 0
- Exec_Master_Log_Pos: 3451
- Relay_Log_Space: 1205
- Until_Condition: None
- Until_Log_File:
- Until_Log_Pos: 0
- Master_SSL_Allowed: No
- Master_SSL_CA_File:
- Master_SSL_CA_Path:
- Master_SSL_Cert:
- Master_SSL_Cipher:
- Master_SSL_Key:
- Seconds_Behind_Master: 0
- Master_SSL_Verify_Server_Cert: No
- Last_IO_Errno: 0
- Last_IO_Error:
- Last_SQL_Errno: 0
- Last_SQL_Error:
- Replicate_Ignore_Server_Ids:
- Master_Server_Id: 27
- Master_UUID: 19a37552-2d44-11e4-af5c-763d1493518d
- Master_Info_File: /usr/local/Percona-Server-5.6.19-rel67.0-618.Linux.x86_64/data/master.info
- SQL_Delay: 0
- SQL_Remaining_Delay: NULL
- Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
- Master_Retry_Count: 86400
- Master_Bind:
- Last_IO_Error_Timestamp:
- Last_SQL_Error_Timestamp:
- Master_SSL_Crl:
- Master_SSL_Crlpath:
- Retrieved_Gtid_Set: 19a37552-2d44-11e4-af5c-763d1493518d:1-10
- Executed_Gtid_Set: 19a37552-2d44-11e4-af5c-763d1493518d:1-10
- Auto_Position: 1