kafka isr, acks を確認 - tjtjtjのメモ

min.insync.replicas=1

トピック作成とトピック確認

bin/kafka-topics.sh --zookeeper 192.168.0.21:2181 \
    --create --topic topic1 \
    --replication-factor 2 --partitions 3 --config min.insync.replicas=1
↓↓↓
Created topic topic1.

bin/kafka-topics.sh --describe --zookeeper 192.168.0.21:2181 --topic topic1
↓↓↓
Topic:topic1    PartitionCount:3        ReplicationFactor:2     Configs:min.insync.replicas=1
        Topic: topic1   Partition: 0    Leader: 2       Replicas: 2,1   Isr: 2,1
        Topic: topic1   Partition: 1    Leader: 3       Replicas: 3,2   Isr: 3,2
        Topic: topic1   Partition: 2    Leader: 1       Replicas: 1,3   Isr: 1,3

プロデュース acks:-1

bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 \
  --topic topic1 \
  --request-required-acks -1
↓↓↓
>a
>b
>c

プロデュース acks:0

bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 \
  --topic topic1 \
  --request-required-acks 0
>d
>e
>f

プロデュース acks:1

bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 \
  --topic topic1 \
  --request-required-acks 1
>g
>h
>i

broker2 停止してトピック確認

bin/kafka-topics.sh --describe --zookeeper 192.168.0.21:2181 --topic topic1
↓↓↓
Topic:topic1    PartitionCount:3        ReplicationFactor:2     Configs:min.insync.replicas=1
        Topic: topic1   Partition: 0    Leader: 1       Replicas: 2,1   Isr: 1
        Topic: topic1   Partition: 1    Leader: 3       Replicas: 3,2   Isr: 3
        Topic: topic1   Partition: 2    Leader: 1       Replicas: 1,3   Isr: 1,3

プロデュース acks:-1

bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 \
  --topic topic1 \
  --request-required-acks -1
↓↓↓
>A
>B
>C

プロデュース acks:0

bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 \
  --topic topic1 \
  --request-required-acks 0
↓↓↓
>D
>E
>F

プロデュース acks:1

bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 \
  --topic topic1 \
  --request-required-acks 1
↓↓↓
>G
>H
>I

コンシューム

bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic topic1 --from-beginning
↓↓↓
a
d
h
C
F
G
b
e
i
A
D
H
c
f
g
B
E
I
^CProcessed a total of 18 messages

min.insync.replicas=2

トピック作成とトピック確認

bin/kafka-topics.sh --zookeeper 192.168.0.21:2181 \
    --create --topic topic2 \
    --replication-factor 2 --partitions 3 --config min.insync.replicas=2
↓↓↓
Created topic topic2.


bin/kafka-topics.sh --describe --zookeeper 192.168.0.21:2181 --topic topic2
↓↓↓
Topic:topic2    PartitionCount:3        ReplicationFactor:2     Configs:min.insync.replicas=2
        Topic: topic2   Partition: 0    Leader: 2       Replicas: 2,3   Isr: 2,3
        Topic: topic2   Partition: 1    Leader: 3       Replicas: 3,1   Isr: 3,1
        Topic: topic2   Partition: 2    Leader: 1       Replicas: 1,2   Isr: 1,2

プロデュース acks:-1

bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 \
  --topic topic2 \
  --request-required-acks -1
↓↓↓
>a
>b
>c

プロデュース acks:0

bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 \
  --topic topic2 \
  --request-required-acks 0
↓↓↓
>d
>e
>f

プロデュース acks:1

bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 \
  --topic topic2 \
  --request-required-acks 1
↓↓↓
>g
>h
>i

broker2 停止してトピック確認。partition:0,2 は isr が1コしかない。acks:-1のとき2/3で失敗するはず。

bin/kafka-topics.sh --describe --zookeeper 192.168.0.21:2181 --topic topic2
↓↓↓
opic:topic2    PartitionCount:3        ReplicationFactor:2     Configs:min.insync.replicas=2
        Topic: topic2   Partition: 0    Leader: 3       Replicas: 2,3   Isr: 3
        Topic: topic2   Partition: 1    Leader: 3       Replicas: 3,1   Isr: 3,1
        Topic: topic2   Partition: 2    Leader: 1       Replicas: 1,2   Isr: 1

プロデュース acks:-1。予想通り A,C の送信に失敗。

bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 \
  --topic topic2 \
  --request-required-acks -1
↓↓↓
>A
>[2019-12-10...] WARN [Producer clientId=console-producer] Got error produce response with correlation id 5 on topic-partition topic2-2, retrying (2 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender)
[2019-12-10...] WARN [Producer clientId=console-producer] Got error produce response with correlation id 6 on topic-partition topic2-2, retrying (1 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender)
[2019-12-10...] WARN [Producer clientId=console-producer] Got error produce response with correlation id 7 on topic-partition topic2-2, retrying (0 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender)
[2019-12-10...] ERROR Error when sending message to topic topic2 with key: null, value: 1 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required.
B
>C
>[2019-12-10...] WARN [Producer clientId=console-producer] Got error produce response with correlation id 10 on topic-partition topic2-0, retrying (2 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender)
[2019-12-10...] WARN [Producer clientId=console-producer] Got error produce response with correlation id 11 on topic-partition topic2-0, retrying (1 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender)
[2019-12-10...] WARN [Producer clientId=console-producer] Got error produce response with correlation id 12 on topic-partition topic2-0, retrying (0 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender)
[2019-12-10...] ERROR Error when sending message to topic topic2 with key: null, value: 1 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required.
^C

プロデュース acks:0

bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 \
  --topic topic2 \
  --request-required-acks 0
↓↓↓
>D
>E
>F

プロデュース acks:1

bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 \
  --topic topic2 \
  --request-required-acks 1
↓↓↓
>G
>H
>I

起動しておいたコンシューマ

bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic topic2 --from-beginning
a
b
c
d
e
f
g
h
i
[2019-12-10...] WARN [Consumer clientId=consumer-1, groupId=console-consumer-71537] Connection to node 2 (/192.168.0.22:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
B
D
E
F
G
H
I
[2019-12-10...] WARN [Consumer clientId=consumer-1, groupId=console-consumer-71537] Connection to node 2 (/192.168.0.22:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
^CProcessed a total of 16 messages