Docker outside of Docker
コンテナからコンテナを操作するその2
コンテナ側からホストのdocker.sock (/var/run/docker.sock)をマウントすることでコンテナ上のDockerコマンドはホスト側のDocker環境で実行される。 https://blog.nijohando.jp/post/docker-in-docker-docker-outside-of-docker/
dood
docker.sock をマウントした docker コンテナをrun してshellに接続
$ docker run -it -v /var/run/docker.sock:/var/run/docker.sock --name some-docker -d docker:stable-dind sh $ docker exec -it some-docker sh
シェルで docker run とかいろいろ試す。ホスト側でいろいろ試した痕跡が見えた。
/ # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a5afae659859 docker:stable-dind "dockerd-entrypoint.…" 19 seconds ago Up 19 seconds 2375-2376/tcp some-docker / # docker run hello-world Hello from Docker! / # docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1a6ced6e0f86 hello-world "/hello" 11 seconds ago Exited (0) 10 seconds ago romantic_curie a5afae659859 docker:stable-dind "dockerd-entrypoint.…" 54 seconds ago Up 53 seconds 2375-2376/tcp some-docker b87a1ce1b0eb docker:stable-dind "dockerd-entrypoint.…" 4 minutes ago Exited (1) 4 minutes ago zealous_engelbart aa633f1665b0 docker:stable-dind "dockerd-entrypoint.…" 4 minutes ago Exited (1) 4 minutes ago gifted_sinoussi 5df11e442065 docker:stable-dind "dockerd-entrypoint.…" 5 minutes ago Exited (1) 4 minutes ago dazzling_sinoussi edb9406c5701 docker:stable-dind "dockerd-entrypoint.…" 6 minutes ago Exited (1) 6 minutes ago peaceful_banzai 5a951b9b9229 hello-world "/hello" 9 minutes ago Exited (0) 9 minutes ago elegant_mclaren 142f3fb5d0bb hello-world "/hello" 9 minutes ago Exited (0) 9 minutes ago gifted_mayer 96114c99440c hello-world "/hello" 11 minutes ago Exited (0) 11 minutes ago xenodochial_archimedes 7563b251264c hello-world "/hello" 12 minutes ago Exited (0) 12 minutes ago vibrant_snyder
Docker in Docker
ちょっと試した。
What is Docker in Docker?
Docker内でDockerを実行することは一般的に推奨されませんが、Docker自体の開発など、正当な使用例がいくつかあります。
dind
dind のイメージをrun。privileged は特権モード
$ docker run -it --privileged --name some-docker -d docker:stable-dind
dind のシェルに接続。
$ docker exec -it some-docker sh
dind のシェルに接続。dind内で docker run とかいろいろ試す。
/ # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES / # docker run hello-world : Hello from Docker! : / # docker images REPOSITORY TAG IMAGE ID CREATED SIZE hello-world latest fce289e99eb9 11 months ago 1.84kB / # docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 474e499442ad hello-world "/hello" 7 minutes ago Exited (0) 7 minutes ago beautiful_perlman 4c897acfba25 hello-world "/hello" 9 minutes ago Exited (0) 9 minutes ago inspiring_saha
ホスト側からdind内は見えない。
$ docker ps -al CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a651d64cd588 docker:stable-dind "dockerd-entrypoint.…" 21 minutes ago Up 21 minutes 2375-2376/tcp some-docker
kafka isr, acks を確認
min.insync.replicas=1
トピック作成とトピック確認
bin/kafka-topics.sh --zookeeper 192.168.0.21:2181 \
--create --topic topic1 \
--replication-factor 2 --partitions 3 --config min.insync.replicas=1
↓↓↓
Created topic topic1.
bin/kafka-topics.sh --describe --zookeeper 192.168.0.21:2181 --topic topic1
↓↓↓
Topic:topic1 PartitionCount:3 ReplicationFactor:2 Configs:min.insync.replicas=1
Topic: topic1 Partition: 0 Leader: 2 Replicas: 2,1 Isr: 2,1
Topic: topic1 Partition: 1 Leader: 3 Replicas: 3,2 Isr: 3,2
Topic: topic1 Partition: 2 Leader: 1 Replicas: 1,3 Isr: 1,3
プロデュース acks:-1
bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 \ --topic topic1 \ --request-required-acks -1 ↓↓↓ >a >b >c
プロデュース acks:0
bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 \ --topic topic1 \ --request-required-acks 0 >d >e >f
プロデュース acks:1
bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 \ --topic topic1 \ --request-required-acks 1 >g >h >i
broker2 停止してトピック確認
bin/kafka-topics.sh --describe --zookeeper 192.168.0.21:2181 --topic topic1
↓↓↓
Topic:topic1 PartitionCount:3 ReplicationFactor:2 Configs:min.insync.replicas=1
Topic: topic1 Partition: 0 Leader: 1 Replicas: 2,1 Isr: 1
Topic: topic1 Partition: 1 Leader: 3 Replicas: 3,2 Isr: 3
Topic: topic1 Partition: 2 Leader: 1 Replicas: 1,3 Isr: 1,3
プロデュース acks:-1
bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 \ --topic topic1 \ --request-required-acks -1 ↓↓↓ >A >B >C
プロデュース acks:0
bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 \ --topic topic1 \ --request-required-acks 0 ↓↓↓ >D >E >F
プロデュース acks:1
bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 \ --topic topic1 \ --request-required-acks 1 ↓↓↓ >G >H >I
コンシューム
bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic topic1 --from-beginning ↓↓↓ a d h C F G b e i A D H c f g B E I ^CProcessed a total of 18 messages
min.insync.replicas=2
トピック作成とトピック確認
bin/kafka-topics.sh --zookeeper 192.168.0.21:2181 \
--create --topic topic2 \
--replication-factor 2 --partitions 3 --config min.insync.replicas=2
↓↓↓
Created topic topic2.
bin/kafka-topics.sh --describe --zookeeper 192.168.0.21:2181 --topic topic2
↓↓↓
Topic:topic2 PartitionCount:3 ReplicationFactor:2 Configs:min.insync.replicas=2
Topic: topic2 Partition: 0 Leader: 2 Replicas: 2,3 Isr: 2,3
Topic: topic2 Partition: 1 Leader: 3 Replicas: 3,1 Isr: 3,1
Topic: topic2 Partition: 2 Leader: 1 Replicas: 1,2 Isr: 1,2
プロデュース acks:-1
bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 \ --topic topic2 \ --request-required-acks -1 ↓↓↓ >a >b >c
プロデュース acks:0
bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 \ --topic topic2 \ --request-required-acks 0 ↓↓↓ >d >e >f
プロデュース acks:1
bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 \ --topic topic2 \ --request-required-acks 1 ↓↓↓ >g >h >i
broker2 停止してトピック確認。partition:0,2 は isr が1コしかない。acks:-1のとき2/3で失敗するはず。
bin/kafka-topics.sh --describe --zookeeper 192.168.0.21:2181 --topic topic2
↓↓↓
opic:topic2 PartitionCount:3 ReplicationFactor:2 Configs:min.insync.replicas=2
Topic: topic2 Partition: 0 Leader: 3 Replicas: 2,3 Isr: 3
Topic: topic2 Partition: 1 Leader: 3 Replicas: 3,1 Isr: 3,1
Topic: topic2 Partition: 2 Leader: 1 Replicas: 1,2 Isr: 1
プロデュース acks:-1。予想通り A,C の送信に失敗。
bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 \ --topic topic2 \ --request-required-acks -1 ↓↓↓ >A >[2019-12-10...] WARN [Producer clientId=console-producer] Got error produce response with correlation id 5 on topic-partition topic2-2, retrying (2 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [2019-12-10...] WARN [Producer clientId=console-producer] Got error produce response with correlation id 6 on topic-partition topic2-2, retrying (1 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [2019-12-10...] WARN [Producer clientId=console-producer] Got error produce response with correlation id 7 on topic-partition topic2-2, retrying (0 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [2019-12-10...] ERROR Error when sending message to topic topic2 with key: null, value: 1 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required. B >C >[2019-12-10...] WARN [Producer clientId=console-producer] Got error produce response with correlation id 10 on topic-partition topic2-0, retrying (2 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [2019-12-10...] WARN [Producer clientId=console-producer] Got error produce response with correlation id 11 on topic-partition topic2-0, retrying (1 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [2019-12-10...] WARN [Producer clientId=console-producer] Got error produce response with correlation id 12 on topic-partition topic2-0, retrying (0 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [2019-12-10...] ERROR Error when sending message to topic topic2 with key: null, value: 1 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required. ^C
プロデュース acks:0
bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 \ --topic topic2 \ --request-required-acks 0 ↓↓↓ >D >E >F
プロデュース acks:1
bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 \ --topic topic2 \ --request-required-acks 1 ↓↓↓ >G >H >I
起動しておいたコンシューマ
bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic topic2 --from-beginning a b c d e f g h i [2019-12-10...] WARN [Consumer clientId=consumer-1, groupId=console-consumer-71537] Connection to node 2 (/192.168.0.22:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) B D E F G H I [2019-12-10...] WARN [Consumer clientId=consumer-1, groupId=console-consumer-71537] Connection to node 2 (/192.168.0.22:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) ^CProcessed a total of 16 messages
kafka コンシューマグループ
コンシューマグループを試した。
あるトピックに複数のコンシューマを接続すると、各コンシューマーが同じメッセージを受信する。 コンシューマグループを使うと同一グループのコンシューマは、同じメッセージを受信しない。同一グループのコンシューマは別パーティションを購読しようとするため。
トピックのパーティション確認
# bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic partitioned-topic
Topic:partitioned-topic PartitionCount:3 ReplicationFactor:1 Configs:
Topic: partitioned-topic Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Topic: partitioned-topic Partition: 1 Leader: 2 Replicas: 2 Isr: 2
Topic: partitioned-topic Partition: 2 Leader: 3 Replicas: 3 Isr: 3
コンシューマ1コ目
# bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --group g1
コンシューマグループ確認。89a2a がパーティション0,1,2 を購読している。
# bin/kafka-consumer-groups.sh --bootstrap-server 192.168.0.21:9092 --group g1 --describe TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID partitioned-topic 0 17 17 0 consumer-1-89a2acb0-b36c-4e92-87b3-abc2a0d27ebe /192.168.0.21 consumer-1 partitioned-topic 1 14 14 0 consumer-1-89a2acb0-b36c-4e92-87b3-abc2a0d27ebe /192.168.0.21 consumer-1 partitioned-topic 2 19 19 0 consumer-1-89a2acb0-b36c-4e92-87b3-abc2a0d27ebe /192.168.0.21 consumer-1
コンシューマ2コ目
# bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.22:9092 --topic partitioned-topic --group g1
コンシューマグループ確認。89a2a がパーティション0,1、ecadがパーティション2を購読している。
# bin/kafka-consumer-groups.sh --bootstrap-server 192.168.0.21:9092 --group g1 --describe TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID partitioned-topic 0 17 17 0 consumer-1-89a2acb0-b36c-4e92-87b3-abc2a0d27ebe /192.168.0.21 consumer-1 partitioned-topic 1 14 14 0 consumer-1-89a2acb0-b36c-4e92-87b3-abc2a0d27ebe /192.168.0.21 consumer-1 partitioned-topic 2 19 19 0 consumer-1-ecad08be-1624-4a48-be03-980b3d5b8ee0 /192.168.0.21 consumer-1
メッセージ送信
この状態で6コのメッセージ送信する。メッセージがラウンドロビンでパーティション配送されるなら 89a2が4つ、ecadが2つ受信するだろう。
# bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 --topic partitioned-topic >a >b >c >d >e >f
コンシューマ1
# bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --group g1 b c e f
コンシューマ2。想定通りの結果
# bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.22:9092 --topic partitioned-topic --group g1 a d
パーティション毎のメッセージ確認
# bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning --partition 0 b e ^CProcessed a total of 2 messages # bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning --partition 1 c f ^CProcessed a total of 2 messages # bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning --partition 2 a d ^CProcessed a total of 2 messages
kafka メッセージの削除
kafka を実験しているとトピックを削除するのでなくメッセージをお掃除したいときがある。kafka-delete-records.shでできるようだ。
削除前の確認
パーティション毎のメッセージ確認
# bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning --partition 0 b e ^CProcessed a total of 2 messages # bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning --partition 1 c f ^CProcessed a total of 2 messages # bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning --partition 2 a d ^CProcessed a total of 2 messages
削除
offsetfile.json を作成。削除というか -1 でoffsetを最後まで進めている。
{"partitions":[{"topic":"partitioned-topic","partition":0,"offset": -1}],"version":1}
# bin/kafka-delete-records.sh --bootstrap-server 192.168.0.21:9092 --offset-json-file offsetfile.json Executing records delete operation Records delete operation completed: partition: partitioned-topic-0 low_watermark: 15
削除後の確認
# bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning --partition 0 ^CProcessed a total of 0 messages # bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning --partition 1 c f ^CProcessed a total of 2 messages # bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning --partition 2 a d ^CProcessed a total of 2 messages # bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning a d c f ^CProcessed a total of 4 messages
参考
kafka レプリケーションパーティションを試す その2
トピック作成
# bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic partitioned-topic
Created topic partitioned-topic.
# bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic partitioned-topic
Topic:partitioned-topic PartitionCount:3 ReplicationFactor:1 Configs:
Topic: partitioned-topic Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Topic: partitioned-topic Partition: 1 Leader: 2 Replicas: 2 Isr: 2
Topic: partitioned-topic Partition: 2 Leader: 3 Replicas: 3 Isr: 3
プロデューサー
# bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 --topic partitioned-topic >a >b >c >d >e >f >g >h >i >j
コンシューマー
broker1 指定。broker 指定してもそのbrokerから取得するわけじゃないのは分かっているが一応。
# bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning b e h a d g j c f i ^CProcessed a total of 10 messages
broker2 指定。
# bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.22:9092 --topic partitioned-topic --from-beginning a d g j b e h c f i ^CProcessed a total of 10 messages
broker3 指定
# bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.23:9092 --topic partitioned-topic --from-beginning a d g j b e h c f i ^CProcessed a total of 10 messages
パーティション指定
# bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning --partition 0 c f i ^CProcessed a total of 3 messages # bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning --partition 1 a d g j ^CProcessed a total of 4 messages # bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning --partition 2 b e h ^CProcessed a total of 3 messages # bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning --partition 3 ^CProcessed a total of 0 messages
broker2 をdownしたらどうなる？
トピック確認。partition:1 の leader:-1 になっている。
# bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic partitioned-topic
Topic:partitioned-topic PartitionCount:3 ReplicationFactor:1 Configs:
Topic: partitioned-topic Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Topic: partitioned-topic Partition: 1 Leader: -1 Replicas: 2 Isr: 2
Topic: partitioned-topic Partition: 2 Leader: 3 Replicas: 3 Isr: 3
コンシューマー。warningしつつ、partition:0,2 分を取得した。
# bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning [2019-12-04 ...] WARN [Consumer clientId=consumer-1, groupId=console-consumer-16999] 1 partitions have leader brokers without a matching listener, including [partitioned-topic-1] (org.apache.kafka.clients.NetworkClient) b e h c f i [2019-12-04 ...] WARN [Consumer clientId=consumer-1, groupId=console-consumer-16999] 1 partitions have leader brokers without a matching listener, including [partitioned-topic-1] (org.apache.kafka.clients.NetworkClient) ^CProcessed a total of 6 messages
プロデューサー。warningしつつ動いた。
bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 --topic partitioned-topic
aa [2019-12-04 ...] WARN [Producer clientId=console-producer] 1 partitions have leader brokers without a matching listener, including [partitioned-topic-1] (org.apache.kafka.clients.NetworkClient) bb cc dd ee ff gg
つづけてコンシューマー。受信できない
bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning
bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.23:9092 --topic partitioned-topic --from-beginning
[2019-12-04 ...] WARN [Consumer clientId=consumer-1, groupId=console-consumer-49753] 1 partitions have leader brokers without a matching listener, including [partitioned-topic-1] (org.apache.kafka.clients.NetworkClient) [2019-12-04 ...] WARN [Consumer clientId=consumer-1, groupId=console-consumer-49753] 1 partitions have leader brokers without a matching listener, including [partitioned-topic-1] (org.apache.kafka.clients.NetworkClient) [2019-12-04 ...] WARN [Consumer clientId=consumer-1, groupId=console-consumer-49753] 1 partitions have leader brokers without a matching listener, including [partitioned-topic-1] (org.apache.kafka.clients.NetworkClient) [2019-12-04 ...] WARN [Consumer clientId=consumer-1, groupId=console-consumer-49753] 1 partitions have leader brokers without a matching listener, including [partitioned-topic-1] (org.apache.kafka.clients.NetworkClient) [2019-12-04 ...] WARN [Consumer clientId=consumer-1, groupId=console-consumer-49753] 1 partitions have leader brokers without a matching listener, including [partitioned-topic-1] (org.apache.kafka.clients.NetworkClient) [2019-12-04 ...] WARN [Consumer clientId=consumer-1, groupId=console-consumer-49753] 1 partitions have leader brokers without a matching listener, including [partitioned-topic-1] (org.apache.kafka.clients.NetworkClient) [2019-12-04 ...] WARN [Consumer clientId=consumer-1, groupId=console-consumer-49753] 1 partitions have leader brokers without a matching listener, including [partitioned-topic-1] (org.apache.kafka.clients.NetworkClient) [2019-12-04 ...] WARN [Consumer clientId=consumer-1, groupId=console-consumer-49753] 1 partitions have leader brokers without a matching listener, including [partitioned-topic-1] (org.apache.kafka.clients.NetworkClient) ^CProcessed a total of 0 messages
■ broker2 をupしたらどうなる？ 欠損はなかった
bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic partitioned-topic
Topic:partitioned-topic PartitionCount:3 ReplicationFactor:1 Configs: Topic: partitioned-topic Partition: 0 Leader: 1 Replicas: 1 Isr: 1 Topic: partitioned-topic Partition: 1 Leader: 2 Replicas: 2 Isr: 2 Topic: partitioned-topic Partition: 2 Leader: 3 Replicas: 3 Isr: 3
bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning
b e h aa cc ee gg c f i bb dd ff a d g j ^CProcessed a total of 17 messages
パーティション指定し確認。broker2 を避けて保存していたようだ。
bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning --partition 0
c f i bb dd ff ^CProcessed a total of 6 messages
bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning --partition 1
a d g j ^CProcessed a total of 4 messages
bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning --partition 2
b e h aa cc ee gg ^CProcessed a total of 7 messages
bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic partitioned-topic --from-beginning --partition 3
^CProcessed a total of 0 messages
kafka レプリケーションパーティションを試す その1
中途半端
トピック作成
# bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 3 --topic replicated-topic Created topic replicated-topic.
トピック確認
list をみる
# bin/kafka-topics.sh --list --zookeeper localhost:2181 replicated-topic
describe を見る。 Partition:0 は リーダーが1。1,2でレプリケーションしている。1,2 とも ISR(In Sync Replica レプリカ済み) 。
# bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic replicated-topic
Topic:replicated-topic PartitionCount:3 ReplicationFactor:2 Configs:
Topic: replicated-topic Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2
Topic: replicated-topic Partition: 1 Leader: 2 Replicas: 2,3 Isr: 2,3
Topic: replicated-topic Partition: 2 Leader: 3 Replicas: 3,1 Isr: 3,1
ブローカー１コ指定で登録/受信
broker1 へメッセージ登録しブローカ1,2,3で受信してみる。順番が異なる場合があるが、全部受信した。
# bin/kafka-console-producer.sh --broker-list 192.168.0.21:9092 --topic replicated-topic >qwer >asdf >zxcv
# bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic replicated-topic --from-beginning asdf zxcv qwer ^CProcessed a total of 3 messages
# bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.22:9092 --topic replicated-topic --from-beginning qwer asdf zxcv ^CProcessed a total of 3 messages
# bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.23:9092 --topic replicated-topic --from-beginning qwer asdf zxcv ^CProcessed a total of 3 messages
brokerをつなげてコンシューム。
# bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092,192.168.0.22:9092,192.168.0.23:9092 --topic replicated-topic --from-beginning asdf zxcv qwer ^CProcessed a total of 3 messages
パーティションを指定してconsume
3メッセージが別パーティションに配置されていたことがわかる。
↓下だと思っていたが、誤りだった。
broker1に登録したメッセージは1,2でレプリケーションする。3からはコンシュームできない。
# bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic replicated-topic --partition 0 --from-beginning zxcv ^CProcessed a total of 1 messages # bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic replicated-topic --partition 1 --from-beginning qwer ^CProcessed a total of 1 messages # bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic replicated-topic --partition 2 --from-beginning asdf ^CProcessed a total of 1 messages
存在しないパーティションの場合
# bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.21:9092 --topic replicated-topic --partition 3 --from-beginning ^CProcessed a total of 0 messages
broker2 からコンシューム。broker1と同じ結果が得られた。
# bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.22:9092 --topic replicated-topic --partition 0 --from-beginning zxcv ^CProcessed a total of 1 messages # bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.22:9092 --topic replicated-topic --partition 1 --from-beginning qwer ^CProcessed a total of 1 messages # bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.22:9092 --topic replicated-topic --partition 2 --from-beginning asdf ^CProcessed a total of 1 messages # bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.22:9092 --topic replicated-topic --partition 3 --from-beginning ^CProcessed a total of 0 messages
broker3 からコンシューム。これも同じ
# bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.23:9092 --topic replicated-topic --partition 0 --from-beginning zxcv ^CProcessed a total of 1 messages # bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.23:9092 --topic replicated-topic --partition 1 --from-beginning qwer ^CProcessed a total of 1 messages # bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.23:9092 --topic replicated-topic --partition 2 --from-beginning asdf ^CProcessed a total of 1 messages # bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.23:9092 --topic replicated-topic --partition 3 --from-beginning ^CProcessed a total of 0 messages
どのブローカーを指定しても、メッセージのパーティションのリーダーを見つけ、リーダーに書き込み/読み出する挙動のなのか。書籍等をあたりたい。
