linux下搭建zookeeper+kafka集群

linux版本：centos7.2

鏈接：https://pan.baidu.com/s/1jjHoz_GQNe9pwTT40ZWryg
提取碼：4mtp

kafka版本：kafka_2.12-2.7.0.tgz

鏈接：https://pan.baidu.com/s/1b5yOLpyPh5rQy9hEGDq2Ag
提取碼：ciai

zookeeper環境:

https://blog.csdn.net/weixin_43914685/article/details/113762882

1.解壓安裝包

[root@slave1 local]# tar -zxvf kafka_2.12-2.7.0.tgz 

2.創建軟連接

[root@slave1 local]# ln -s kafka_2.12-2.7.0 kafka0
[root@slave1 local]# ls
apache-tomcat-8.5.63         bin  games    kafka0            kafka_2.12-2.7.0.tgz  lib64    lz-server  share     src
apache-tomcat-8.5.63.tar.gz  etc  include  kafka_2.12-2.7.0  lib                   libexec  sbin       software

3.修改kafka組態檔

[root@master local]# vim kafka0/config/server.properties 

3臺機器分別修改id為0 1 2，listeners修改為自己的ip地址

broker.id=0

############################# Socket Server Settings #############################

# The address the socket server listens on. It will get the value returned from 
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://192.168.1.108:9092

4.日志生成目錄

log.dirs=/tmp/kafka-logs

5.設定Zookeeper集群地址

zookeeper.connect=192.168.1.108:2181,192.168.1.109:2181,192.168.1.110:2181

6.Topic的默認Partition數量

num.partitions 為新建Topic的默認Partition數量，partition數量提升，一定程度上可以提升并發性

num.partitions=3

offsets和transaction
兩個topic，分組元資料的復制因子，為了保證可用性，在生產上建議設定大于1，

default.replication.factor為kafka保存訊息的副本數，如果一個副本失效了，另一個還可以繼續提供服務，是在自動創建topic時的默認副本數，可以設定為3

############################# Internal Topic Settings  #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=3

7.啟動kafka

[root@slave2 kafka0]# bin/kafka-server-start.sh -daemon config/server.properties

8.查看日志

進入kafka的log目錄,結尾輸出日志started 啟動成功

[root@slave2 logs]# cd logs/
[root@slave2 logs]# cat kafkaServer.out 
[2021-02-13 06:48:06,563] INFO [KafkaServer id=2] started (kafka.server.KafkaServer)
[2021-02-13 06:48:09,168] INFO [broker-2-to-controller-send-thread]: Recorded new controller, from now on will use broker 2 (kafka.server.BrokerToControllerRequestThread)

9.查看行程

[root@slave2 logs]# jps
2768 Jps
2248 QuorumPeerMain
2682 Kafka

10.停止kafka命令

[root@slave2 logs]# bin/kafka-server-stop.sh config/server.properties

11.配置環境變數

[root@master ~]# vim /etc/profile
[root@master ~]# source /etc/profile
[root@master ~]# echo $KAFKA_HOME

12.kafka和Zookeeper已啟動完成

1. 創建topic集群，3臺機器可以同步接收訊息，一臺掛掉不影響另外兩臺繼續作業

[root@master ~]# kafka-topics.sh --create --zookeeper '192.168.1.108:2181,192.168.1.109:2181,192.168.1.110:2181' --replication-factor 3 --partitions 3 --topic kafka0
Created topic kafka0.

1. 查看主題

[root@master ~]# kafka-topics.sh --list --zookeeper 192.168.1.108:2181
__consumer_offsets
kafka0

2. 發送訊息

通過192.168.1.109給192.168.1.108發送訊息

[root@master kafka1]# bin/kafka-console-producer.sh --broker-list 192.168.1.108:9092 --topic test

3. 接收訊息

192.168.1.108主機接收訊息

[root@master ~]# kafka-console-consumer.sh --bootstrap-server 192.168.1.108:9092 --topic kafka0 --from-beginning
hello

4. 查看主題

[root@master ~]# kafka-topics.sh --zookeeper localhost:2181 --describe  --topic kafka0
Topic: kafka0   PartitionCount: 3       ReplicationFactor: 3    Configs: 
        Topic: kafka0   Partition: 0    Leader: 1       Replicas: 1,2,0 Isr: 1,2,0
        Topic: kafka0   Partition: 1    Leader: 2       Replicas: 2,0,1 Isr: 2,0,1
        Topic: kafka0   Partition: 2    Leader: 0       Replicas: 0,1,2 Isr: 0,1,2

從中可以看到，test主題分了三個區，復制因子是3，

5. 洗掉主題

[root@master ~]# kafka-topics.sh --zookeeper localhost:2181 --delete  --topic test
Topic test is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.