tjtjtjのメモ

自分のためのメモです

kafka レプリケーションパーティションを試す その2

パーティションするがレプリケーションしない場合を試した。

トピック作成

# bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic partitioned-topic
Created topic partitioned-topic.
# bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic partitioned-topic
Topic:partitioned-topic PartitionCount:3        ReplicationFactor:1     Configs:
        Topic: partitioned-topic        Partition: 0    Leader: 1       Replicas: 1     Isr: 1
        Topic: partitioned-topic        Partition: 1    Leader: 2       Replicas: 2     Isr: 2
        Topic: partitioned-topic        Partition: 2    Leader: 3       Replicas: 3     Isr: 3

プロデューサー

# bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 --topic partitioned-topic
>a
>b
>c
>d
>e
>f
>g
>h
>i
>j

コンシューマー

broker1 指定。broker 指定してもそのbrokerから取得するわけじゃないのは分かっているが一応。

# bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning
b
e
h
a
d
g
j
c
f
i
^CProcessed a total of 10 messages

broker2 指定。

# bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.22:9092 --topic partitioned-topic --from-beginning
a
d
g
j
b
e
h
c
f
i
^CProcessed a total of 10 messages

broker3 指定

# bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.23:9092 --topic partitioned-topic --from-beginning
a
d
g
j
b
e
h
c
f
i
^CProcessed a total of 10 messages

パーティション指定

# bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning --partition 0
c
f
i
^CProcessed a total of 3 messages
# bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning --partition 1
a
d
g
j
^CProcessed a total of 4 messages
# bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning --partition 2
b
e
h
^CProcessed a total of 3 messages
# bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning --partition 3
^CProcessed a total of 0 messages

broker2 をdownしたらどうなる？

トピック確認。partition:1 の leader:-1 になっている。

# bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic partitioned-topic
Topic:partitioned-topic PartitionCount:3        ReplicationFactor:1     Configs:
        Topic: partitioned-topic        Partition: 0    Leader: 1       Replicas: 1     Isr: 1
        Topic: partitioned-topic        Partition: 1    Leader: -1      Replicas: 2     Isr: 2
        Topic: partitioned-topic        Partition: 2    Leader: 3       Replicas: 3     Isr: 3

コンシューマー。warningしつつ、partition:0,2 分を取得した。

# bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning
[2019-12-04 ...] WARN [Consumer clientId=consumer-1, groupId=console-consumer-16999] 1 partitions have leader brokers without a matching listener, including [partitioned-topic-1] (org.apache.kafka.clients.NetworkClient)
b
e
h
c
f
i
[2019-12-04 ...] WARN [Consumer clientId=consumer-1, groupId=console-consumer-16999] 1 partitions have leader brokers without a matching listener, including [partitioned-topic-1] (org.apache.kafka.clients.NetworkClient)
^CProcessed a total of 6 messages

プロデューサー。warningしつつ動いた。

bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 --topic partitioned-topic

aa [2019-12-04 ...] WARN [Producer clientId=console-producer] 1 partitions have leader brokers without a matching listener, including [partitioned-topic-1] (org.apache.kafka.clients.NetworkClient) bb cc dd ee ff gg

つづけてコンシューマー。受信できない

bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning

bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.23:9092 --topic partitioned-topic --from-beginning

[2019-12-04 ...] WARN [Consumer clientId=consumer-1, groupId=console-consumer-49753] 1 partitions have leader brokers without a matching listener, including [partitioned-topic-1] (org.apache.kafka.clients.NetworkClient) [2019-12-04 ...] WARN [Consumer clientId=consumer-1, groupId=console-consumer-49753] 1 partitions have leader brokers without a matching listener, including [partitioned-topic-1] (org.apache.kafka.clients.NetworkClient) [2019-12-04 ...] WARN [Consumer clientId=consumer-1, groupId=console-consumer-49753] 1 partitions have leader brokers without a matching listener, including [partitioned-topic-1] (org.apache.kafka.clients.NetworkClient) [2019-12-04 ...] WARN [Consumer clientId=consumer-1, groupId=console-consumer-49753] 1 partitions have leader brokers without a matching listener, including [partitioned-topic-1] (org.apache.kafka.clients.NetworkClient) [2019-12-04 ...] WARN [Consumer clientId=consumer-1, groupId=console-consumer-49753] 1 partitions have leader brokers without a matching listener, including [partitioned-topic-1] (org.apache.kafka.clients.NetworkClient) [2019-12-04 ...] WARN [Consumer clientId=consumer-1, groupId=console-consumer-49753] 1 partitions have leader brokers without a matching listener, including [partitioned-topic-1] (org.apache.kafka.clients.NetworkClient) [2019-12-04 ...] WARN [Consumer clientId=consumer-1, groupId=console-consumer-49753] 1 partitions have leader brokers without a matching listener, including [partitioned-topic-1] (org.apache.kafka.clients.NetworkClient) [2019-12-04 ...] WARN [Consumer clientId=consumer-1, groupId=console-consumer-49753] 1 partitions have leader brokers without a matching listener, including [partitioned-topic-1] (org.apache.kafka.clients.NetworkClient) ^CProcessed a total of 0 messages

■ broker2 をupしたらどうなる？

欠損はなかった

bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic partitioned-topic

Topic:partitioned-topic PartitionCount:3 ReplicationFactor:1 Configs: Topic: partitioned-topic Partition: 0 Leader: 1 Replicas: 1 Isr: 1 Topic: partitioned-topic Partition: 1 Leader: 2 Replicas: 2 Isr: 2 Topic: partitioned-topic Partition: 2 Leader: 3 Replicas: 3 Isr: 3

bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning

b e h aa cc ee gg c f i bb dd ff a d g j ^CProcessed a total of 17 messages

パーティション指定し確認。broker2 を避けて保存していたようだ。

bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning --partition 0

c f i bb dd ff ^CProcessed a total of 6 messages

bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning --partition 1

a d g j ^CProcessed a total of 4 messages

bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning --partition 2

b e h aa cc ee gg ^CProcessed a total of 7 messages

bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning --partition 3

^CProcessed a total of 0 messages