cloudera cdh配置kafka

🧙 Questions

默认的cdh6的kafka组件，无法正常使用

☄️ Ideas

配置zookeeper.chroot

1. zookeeper.chroot: /kafka

   将偏移量交给zk管理

2. offsets.topic.replication.factor: 1

   默认是3，注意创建topic保持数量一致，或者直接把值改为1

3. kafka.properties

   内网ip: 223.166.23.72
   外网ip: 192.168.115.104

listeners=PLAINTEXT://172.23.39.236:30196
advertised.listeners=PLAINTEXT://192.168.115.104:30196

常用命令

# 创建topic
kafka-topics --create --zookeeper 192.168.115.104:30121/kafka --topic ispong-topic3 --replication-factor 1 --partitions 1 

# 生产topic
kafka-console-producer --broker-list 192.168.115.104:30196 --topic ispong-topic3

# 消费topic
kafka-console-consumer --bootstrap-server 192.168.115.104:30196 --topic dbserver1.ispong_db.users

# 删除topic
kafka-topics --delete --zookeeper 192.168.115.104:30121/kafka --topic ispong-topic

# 查询所有的topic
kafka-topics --list --zookeeper 192.168.115.104:30121/kafka

删除topic

zookeeper-client -server 192.168.115.104:30121
ls /kafka/brokers/topics
rmr /kafka/brokers/topics/ispong-topic2

🔗 Links

• hadoop docs
• kafka cdh