Amoeba+Mysql實作資料庫讀寫分離
一、Amoeba 是什么
Amoeba(變形蟲)專案,專注 分布式資料庫 proxy 開發,座落與Client、DB Server(s)之間,對客戶端透明,具有負載均衡、高可用性、sql過濾、讀寫分離、可路由相關的query到目標資料庫、可并發請求多臺資料庫合并結果,
主要解決:
? 降低 資料切分帶來的復雜多資料庫結構
? 提供切分規則并降低 資料切分規則 給應用帶來的影響
? 降低db 與客戶端的連接數
? 讀寫分離
二、為什么要用Amoeba
目前要實作mysql的主從讀寫分離,主要有以下幾種方案:
1、 通程序式實作,網上很多現成的代碼,比較復雜,如果添加從服務器要更改多臺服務器的代碼,
2、 通過mysql-proxy來實作,由于mysql-proxy的主從讀寫分離是通過lua腳本來實作,目前lua的腳本的開發跟不上節奏,而寫沒有完美的現成的腳本,因此導致用于生產環境的話風險比較大,據網上很多人說mysql-proxy的性能不高,
3、 自己開發介面實作,這種方案門檻高,開發成本高,不是一般的小公司能承擔得起,
4、 利用阿里巴巴的開源專案Amoeba來實作,具有負載均衡、高可用性、sql過濾、讀寫分離、可路由相關的query到目標資料庫,并且安裝配置非常簡單,國產的開源軟體,應該支持,目前正在使用,不發表太多結論,一切等測驗完再發表結論吧,哈哈!

三、Amoeba+Mysql實作資料庫讀寫分離
基本的原理:讓“主”資料庫處理事務增,刪,改(INSERT,UPDATA,DELETE),“從”資料庫處理SELECT查詢操作,資料庫復制被用來把事務性操作導致變更同步到集群中的從資料庫
先介紹下部署環境:
amoeba(代理服務器):192.168.220.33
master-mysql:192.168.220.30
slave1-mysql:192.168.220.31
slave2-mysql:192.168.220.32
客戶機(本機 win11):192.168.220.2
系統:centOS 7.9
MySQL:5.7
Amoeba:3.0.5 (框架是居于java語言開發)
jdk:1.8
部署MySQL一主多從:
注:做 主從 資料庫里資料最好是保持一致
1、主資料庫:master-mysql
mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | sys | +--------------------+ 4 rows in set (0.00 sec) mysql> mysql> grant all on *.* to 'test'@'192.168.220.%' identified by '1234.Com'; #創建授權用戶 為amoeba提供登錄 Query OK, 0 rows affected, 1 warning (0.00 sec) mysql> mysql> grant replication slave on *.* to 'slave'@'192.168.220.%' identified by '1234.Com'; #主從模式 Query OK, 0 rows affected, 1 warning (0.00 sec) mysql> mysql> show master status\G; *************************** 1. row *************************** File: mysql-bin.000001 #從庫加入主庫時 需要配置的二進制檔案名 Position: 154 # 二進制日子 從 pos 154 開始被slave 復制 備份 Binlog_Do_DB: Binlog_Ignore_DB: Executed_Gtid_Set: 1 row in set (0.00 sec) ERROR: No query specified mysql>
2、從資料庫:slave1-mysql
mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | sys | +--------------------+ 4 rows in set (0.00 sec) mysql> mysql> change master to master_host='192.168.220.30',master_user='slave',master_password='1234.Com',,master_log_file='mysql-bin.000001',master_log_pos=154; Query OK, 0 rows affected, 2 warnings (0.01 sec) mysql> mysql> start slave; Query OK, 0 rows affected (0.01 sec) mysql> show slave status\G; #查看從庫的連接狀態,I/O,SQL 都要為 YES *************************** 1. row *************************** Slave_IO_State: Connecting to master Master_Host: 192.168.220.30 Master_User: slave Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 154 Relay_Log_File: slave1-relay-bin.000001 Relay_Log_Pos: 4 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: YES Slave_SQL_Running: Yes Replicate_Do_DB: ···················· ERROR: No query specified
mysql> grant all on *.* to 'test'@'192.168.220.%' identified by '1234.Com'; #創建授權用戶 為amoeba提供登錄
Query OK, 0 rows affected, 1 warning (0.00 sec)
3、從資料庫:slave2-mysql
mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | sys | +--------------------+ 4 rows in set (0.00 sec)
mysql> mysql> change master to master_host='192.168.220.30',master_user='slave',master_password='1234.Com',,master_log_file='mysql-bin.000001',master_log_pos=154; Query OK, 0 rows affected, 2 warnings (0.01 sec) mysql> mysql> start slave; Query OK, 0 rows affected (0.01 sec) mysql> show slave status\G; #查看從庫的連接狀態,I/O,SQL 都要為 YES *************************** 1. row *************************** Slave_IO_State: Connecting to master Master_Host: 192.168.220.30 Master_User: slave Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 154 Relay_Log_File: slave1-relay-bin.000001 Relay_Log_Pos: 4 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: YES Slave_SQL_Running: Yes Replicate_Do_DB: ···················· ERROR: No query specified
mysql> grant all on *.* to 'test'@'192.168.220.%' identified by '1234.Com'; #創建授權用戶 為amoeba提供登錄
Query OK, 0 rows affected, 1 warning (0.00 sec)
4、主庫測驗:創建一個測驗庫并添加創建表和添加資料,查看從庫是否同步
mysql> create database test; Query OK, 1 row affected (0.00 sec) mysql> use test; Database changed mysql> create table test (id int); Query OK, 0 rows affected (0.01 sec) mysql> insert into test values (1); Query OK, 1 row affected (0.00 sec) mysql> select * from test; +------+ | id | +------+ | 1 | +------+ 1 row in set (0.00 sec) mysql>
5、slave1 從庫測驗:
mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | sys | | test | +--------------------+ 5 rows in set (0.00 sec) mysql> select * from test.test; +------+ | id | +------+ | 1 | +------+ 1 row in set (0.00 sec) mysql>
6、slave2 從庫測驗:
mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | sys | | test | +--------------------+ 5 rows in set (0.00 sec) mysql> select * from test.test; +------+ | id | +------+ | 1 | +------+ 1 row in set (0.00 sec) mysql>
從資料庫同步正常,主從資料庫部署完成,接下來部署 Amoeba
Amoeba 安裝部署
1、安裝java環境
先去官網下載:http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
安裝
[root@amoeba src]# rpm -ivh jdk-8u111-linux-x64.rpm Preparing... ########################################### [100%] 1:jdk1.8.0_111 ########################################### [100%] Unpacking JAR files... tools.jar... plugin.jar... javaws.jar... deploy.jar... rt.jar... jsse.jar... charsets.jar... localedata.jar...
設定Java環境
[root@amoeba src]# vim /etc/profile #set java environment JAVA_HOME=/usr/java/jdk1.8.0_111 JRE_HOME=/usr/java/jdk1.8.0_111/jre CLASS_PATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin export JAVA_HOME JRE_HOME CLASS_PATH PATH[root@bogon amoeba]# source /etc/profile
測驗是否安裝成功
[root@bogon src]# java -version java version "1.8.0_111" Java(TM) SE Runtime Environment (build 1.8.0_111-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
2、安裝Amoeba
可以從https://sourceforge.net/projects/amoeba/files/Amoeba%20for%20mysql/3.x/下載最新版本的Amoeba,我這里下載的是amoeba-mysql-3.0.5-RC-distribution.zip,Amoeba安裝非常簡單,直接解壓即可使用,這里將Amoeba解壓到/usr/local/amoeba目錄下,這樣就安裝完成了
[root@amoeba amoeba]# pwd /usr/local/amoeba [root@amoeba amoeba]# ll total 48 drwxr-xr-x 2 root root 63 Aug 16 10:24 benchmark drwxr-xr-x 2 root root 131 Feb 29 2012 bin -rwxr-xr-x 1 root root 3976 Aug 29 2012 changelogs.txt drwxr-xr-x 2 root root 243 Aug 16 14:53 conf drwxr-xr-x 3 root root 4096 Aug 16 10:24 lib -rwxr-xr-x 1 root root 34520 Aug 29 2012 LICENSE.txt drwxr-xr-x 2 root root 133 Aug 16 10:25 logs -rwxr-xr-x 1 root root 2031 Aug 29 2012 README.html [root@amoeba amoeba]#
3.配置Amoeba(實作讀寫分離,兩個slave“讀”負載)
Amoeba的組態檔在本環境下位于:/usr/local/amoeba/conf 目錄下,實作讀寫分離和負載 只需要 dbServers.xml和amoeba.xml 兩個檔案即可,下面介紹
dbServer.xml
[root@amoeba amoeba]# cd conf/ [root@amoeba conf]# ls access_list.conf amoeba.xml dbServers.xml functionMap.xml log4j.xml ruleFunctionMap.xml amoeba.dtd dbserver.dtd function.dtd log4j.dtd rule.dtd rule.xml [root@amoeba conf]# cat dbServers.xml <?xml version="1.0" encoding="gbk"?> <!DOCTYPE amoeba:dbServers SYSTEM "dbserver.dtd"> <amoeba:dbServers xmlns:amoeba="http://amoeba.meidusa.com/"> <!-- Each dbServer needs to be configured into a Pool, If you need to configure multiple dbServer with load balancing that can be simplified by the following configuration: add attribute with name virtual = "true" in dbServer, but the configuration does not allow the element with name factoryConfig such as 'multiPool' dbServer --> <dbServer name="abstractServer" abstractive="true"> <factoryConfig class="com.meidusa.amoeba.mysql.net.MysqlServerConnectionFactory"> <property name="manager">${defaultManager}</property> <property name="sendBufferSize">64</property> <property name="receiveBufferSize">128</property> <!-- mysql port --> <property name="port">3306</property> #設定Amoeba 連接資料庫的埠,默認 3306 <!-- mysql schema --> <property name="schema">mysql</property> #設定 預設的資料庫,我這里使用的MySQL版本為5.7版本,默認是沒有test庫的,會報所以找不到默認的庫,我之前搭建的MySQL版本是5.5是不需要修改的,沒注意到這一點 <!-- mysql user --> <property name="user">test</property> #設定amoeba連接后端資料庫服務器的賬號和密碼,上面已為 test 授權 和設定了密碼 <!-- mysql password --> <property name="password">1234.Com</property> </factoryConfig> <poolConfig class="com.meidusa.amoeba.net.poolable.PoolableObjectPool"> <property name="maxActive">500</property> #最大連接數,默認為500 <property name="maxIdle">500</property> #最大空閑連接數 <property name="minIdle">10</property> #最新空閑連接數 <property name="minEvictableIdleTimeMillis">600000</property> <property name="timeBetweenEvictionRunsMillis">600000</property> <property name="testOnBorrow">true</property> <property name="testOnReturn">true</property> <property name="testWhileIdle">true</property> </poolConfig> </dbServer> <dbServer name="master" parent="abstractServer"> #設定一個空間名稱,可任意命名,這里定義為:master,顧名思義就是為master庫創建一個命名空間,后面會用到 <factoryConfig> <!-- mysql ip --> <property name="ipAddress">192.168.220.30</property> #設定 master的IP,功能可寫 </factoryConfig> </dbServer> <dbServer name="slave1" parent="abstractServer"> #設定slava1 命名空間名稱 <factoryConfig> <!-- mysql ip --> <property name="ipAddress">192.168.220.31</property> # 設定slave1的IP,功能可寫 </factoryConfig> </dbServer> <dbServer name="slave2" parent="abstractServer"> #設定slava2 命名空間名稱 <factoryConfig> <!-- mysql ip --> <property name="ipAddress">192.168.220.32</property> #設定 slave的ip,功能可寫 </factoryConfig> </dbServer> <dbServer name="slaves" virtual="true"> #設定定義一個虛擬的dbserver,實際上相當于一個dbserver組,這里將可讀的資料庫ip統一放到一個組中,將這個組的名字命名為slaves <poolConfig class="com.meidusa.amoeba.server.MultipleServerPool"> <!-- Load balancing strategy: 1=ROUNDROBIN , 2=WEIGHTBASED , 3=HA--> <property name="loadbalance">1</property> #選擇調度演算法,1表示復制均衡,2表示權重,3表示HA, 這里選擇1 <!-- Separated by commas,such as: server1,server2,server1 --> <property name="poolNames">slave1,slave2</property> #slave組成員,把slave1,slave2 加入成員里,可實作 “讀” 負載 </poolConfig> </dbServer> </amoeba:dbServers> [root@amoeba conf]#
amoeba.xml
[root@amoeba amoeba]# cd conf/ [root@amoeba conf]# ls access_list.conf amoeba.xml dbServers.xml functionMap.xml log4j.xml ruleFunctionMap.xml amoeba.dtd dbserver.dtd function.dtd log4j.dtd rule.dtd rule.xml [root@amoeba conf]# cat amoeba.xml <?xml version="1.0" encoding="gbk"?> <!DOCTYPE amoeba:configuration SYSTEM "amoeba.dtd"> <amoeba:configuration xmlns:amoeba="http://amoeba.meidusa.com/"> <proxy> <!-- service class must implements com.meidusa.amoeba.service.Service --> <service name="Amoeba for Mysql" class="com.meidusa.amoeba.net.ServerableConnectionManager"> <!-- port --> <property name="port">8066</property> #設定amoeba 今天的埠,默認是8066 <!-- bind ipAddress --> <!-- <property name="ipAddress">127.0.0.1</property> --> <property name="manager">${clientConnectioneManager}</property> <property name="connectionFactory"> <bean class="com.meidusa.amoeba.mysql.net.MysqlClientConnectionFactory"> <property name="sendBufferSize">128</property> <property name="receiveBufferSize">64</property> </bean> </property> <property name="authenticator"> <bean class="com.meidusa.amoeba.mysql.server.MysqlClientAuthenticator"> <property name="user">amoeba</property> #提供客服端連接amoeba是需要使用設定的賬號,賬號密碼可任意設定和資料服務器里的密碼無關 <property name="password">123456</property> <property name="filter"> <bean class="com.meidusa.amoeba.server.IPAccessController"> <property name="ipFile">${amoeba.home}/conf/access_list.conf</property> </bean> </property> </bean> </property> </service> <!-- server class must implements com.meidusa.amoeba.service.Service --> <service name="Amoeba Monitor Server" class="com.meidusa.amoeba.monitor.MonitorServer"> <!-- port --> <!-- default value: random number <property name="port">9066</property> --> <!-- bind ipAddress --> <property name="ipAddress">127.0.0.1</property> <property name="daemon">true</property> <property name="manager">${clientConnectioneManager}</property> <property name="connectionFactory"> <bean class="com.meidusa.amoeba.monitor.net.MonitorClientConnectionFactory"></bean> </property> </service> <runtime class="com.meidusa.amoeba.mysql.context.MysqlRuntimeContext"> <!-- proxy server net IO Read thread size --> <property name="readThreadPoolSize">20</property> <!-- proxy server client process thread size --> <property name="clientSideThreadPoolSize">30</property> <!-- mysql server data packet process thread size --> <property name="serverSideThreadPoolSize">30</property> <!-- per connection cache prepared statement size --> <property name="statementCacheSize">500</property> <!-- query timeout( default: 60 second , TimeUnit:second) --> <property name="queryTimeout">60</property> </runtime> </proxy> <!-- Each ConnectionManager will start as thread manager responsible for the Connection IO read , Death Detection --> <connectionManagerList> <connectionManager name="clientConnectioneManager" class="com.meidusa.amoeba.net.MultiConnectionManagerWrapper"> <property name="subManagerClassName">com.meidusa.amoeba.net.ConnectionManager</property> <!-- default value is avaliable Processors <property name="processors">5</property> --> </connectionManager> <connectionManager name="defaultManager" class="com.meidusa.amoeba.net.MultiConnectionManagerWrapper"> <property name="subManagerClassName">com.meidusa.amoeba.net.AuthingableConnectionManager</property> <!-- default value is avaliable Processors <property name="processors">5</property> --> </connectionManager> </connectionManagerList> <!-- default using file loader --> <dbServerLoader class="com.meidusa.amoeba.context.DBServerConfigFileLoader"> <property name="configFile">${amoeba.home}/conf/dbServers.xml</property> </dbServerLoader> <queryRouter class="com.meidusa.amoeba.mysql.parser.MysqlQueryRouter"> <property name="ruleLoader"> <bean class="com.meidusa.amoeba.route.TableRuleFileLoader"> <property name="ruleFile">${amoeba.home}/conf/rule.xml</property> <property name="functionFile">${amoeba.home}/conf/ruleFunctionMap.xml</property> </bean> </property> <property name="sqlFunctionFile">${amoeba.home}/conf/functionMap.xml</property> <property name="LRUMapSize">1500</property> <property name="defaultPool">master</property> #設定amoeba默認是的池,這里設定為master(在dbServers.xml檔案中,我們定義了一會master的空間名稱以及對應的服務器IP) <property name="writePool">master</property> #設定寫的池,master作為主資料庫,處理的業務為 增,刪,改 都是寫操作,所以這設定為master <property name="readPool">slaves</property> #設定讀的池 slave 作為從庫,負責 查(SELECT,因為我們有兩個slave,在定義命名空間時,我們配置了一個slaves組,里面有slave1,slave2兩個成員,這樣就可以達到 讀的負載 <property name="needParse">true</property> </queryRouter> </amoeba:configuration>
3、啟動Amoeba
[root@amoeba ~]# /usr/local/amoeba/bin/launcher & at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:329) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:239) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409) at org.codehaus.classworlds.Launcher.mainWithExitCode(Launcher.java:127) at org.codehaus.classworlds.Launcher.main(Launcher.java:110) Caused by: com.meidusa.toolkit.common.bean.util.InitialisationException: default pool required!,defaultPool=writedb invalid at com.meidusa.amoeba.route.AbstractQueryRouter.init(AbstractQueryRouter.java:469) at com.meidusa.amoeba.context.ProxyRuntimeContext.initAllInitialisableBeans(ProxyRuntimeContext.java:337) ... 11 more 2022-8-16 18:46:37 [INFO] Project Name=Amoeba-MySQL, PID=1577 , System shutdown .... Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=16m; support was removed in 8.0 Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=96m; support was removed in 8.0 2022-8-16 18:50:19 [INFO] Project Name=Amoeba-MySQL, PID=1602 , starting... log4j:WARN log4j config load completed from file:/usr/local/amoeba/conf/log4j.xml 2022-8-16 18:50:21,668 INFO context.MysqlRuntimeContext - Amoeba for Mysql current versoin=5.1.45-mysql-amoeba-proxy-3.0.4-BETA log4j:WARN ip access config load completed from file:/usr/local/amoeba/conf/access_list.conf 2022-8-16 18:50:22,852 INFO net.ServerableConnectionManager - Server listening on 0.0.0.0/0.0.0.0:8066.
查看埠
[root@amoeba ~]# netstat -anpt | grep java tcp6 0 0 127.0.0.1:13319 :::* LISTEN 21350/java tcp6 0 0 :::8066 :::* LISTEN 21350/java
4、Amoeba 登錄測驗(客戶端-本機:win11)

測驗結果:登錄后 所在的庫是“mysql”庫,可讀取,可寫入,到這里資料庫讀寫分離部,負載部署完成!!!
下面就是讀寫分離的效果測驗:
四、測驗讀寫分離,負載效果
1、讀寫分離測驗:
測驗1 :mysql-master down機,寫入報錯,讀正常
mysql-master:
[root@mysql_master ~]# systemctl stop mysqld
客服端:
mysql> select * from test; #可正常查詢 +------+ | id | +------+ | 1 | | 2 | +------+ 2 rows in set (0.00 sec) mysql> mysql> insert into test values (3); ERROR 2013 (HY000): Lost connection to MySQL server during query #寫入失敗
測驗2 :關閉兩臺slave,讀取失敗,插入成功
mysql-master
[root@mysql_master ~]# systemctl start mysqld
mysql-slave1
[root@slave1 ~]# systemctl stop mysqld
mysql-slave2
[root@slave2 ~]# systemctl stop mysqld
客服端:
mysql> insert into test values (3); Query OK, 1 row affected (0.00 sec) mysql> select * from test; ERROR 2013 (HY000): Lost connection to MySQL server during query #查詢失敗 mysql>
測驗3:開啟slava1,save2上的mysql,查看資料是否自動同步
mysql-slave1
[root@slave1 ~]# systemctl start mysqld
mysql-slave2
[root@slave2 ~]# systemctl start mysqld
客服端:
mysql> select * from test; +------+ | id | +------+ | 1 | | 2 | | 3 | +------+ 4 rows in set (0.00 sec) mysql> mysql> insert into test values (4); Query OK, 1 row affected (0.00 sec) mysql> select * from test; +------+ | id | +------+ | 1 | | 2 | | 3 | | 4 | +------+ 5 rows in set (0.00 sec) mysql>讀寫分離測驗完畢!!!
2、slave “讀” 負載測驗:
mysql-slave1:在test表插入值:100
[root@slave1 ~]# mysql -uroot -p Enter password: mysql> mysql> select * from test; +------+ | id | +------+ | 1 | | 2 | | 3 | | 4 | +------+ 4 rows in set (0.00 sec) mysql> insert into test values (100); Query OK, 1 row affected (0.00 sec) mysql> select * from test; +------+ | id | +------+ | 1 | | 2 | | 3 | | 4 | | 100 | +------+ 5 rows in set (0.00 sec) mysql>
mysql-slave2:在test表插入值:200
[root@slave2 ~]# mysql -uroot -p Enter password: mysql> mysql> select * from test; +------+ | id | +------+ | 1 | | 2 | | 3 | | 4 | +------+ 4 rows in set (0.00 sec) mysql> insert into test values (200); Query OK, 1 row affected (0.00 sec) mysql> select * from test; +------+ | id | +------+ | 1 | | 2 | | 3 | | 4 | | 200 | +------+ 5 rows in set (0.00 sec) mysql>
客服端:
mysql> select * from test; +------+ | id | +------+ | 1 | | 2 | | 3 | | 4 | | 200 | +------+ 5 rows in set (0.00 sec) mysql> select * from test; +------+ | id | +------+ | 1 | | 2 | | 3 | | 4 | | 100 | +------+ 5 rows in set (0.00 sec) mysql> select * from test; +------+ | id | +------+ | 1 | | 2 | | 3 | | 4 | | 200 | +------+ 5 rows in set (0.00 sec) mysql> select * from test; +------+ | id | +------+ | 1 | | 2 | | 3 | | 4 | | 100 | +------+ 5 rows in set (0.00 sec) mysql>
分別在 slave1,slava2 上對test表插入不同的資料,通過客戶端查詢 會發現,每一次查詢和上一次不同,這里就是完全體系了 “讀 ”負載的功能,
五、可能會報錯:
在使用amoeba連接資料庫時,報錯java.lang.Exception: poolName=slaves, no valid pools
到這里 對資料庫的高可用架構實驗到此結束!!!
本文來自博客園,作者:xiao智,轉載請注明原文鏈接:https://www.cnblogs.com/yuwen01/p/16591876.html
轉載請註明出處,本文鏈接:https://www.uj5u.com/shujuku/502073.html
標籤:其他
下一篇:Redis主從復制
