kubernetes 学習 logging その1
Logging Architecture を試す。
- コンテナ化されたアプリケーションの最も簡単で包括的なロギング方法は、標準出力と標準エラーストリームに書き込むこと
- クラスタレベルのログ記録では、ログを保存、分析、およびクエリするために別のバックエンドが必要
- Kubernetesはログデータ用のネイティブストレージソリューションを提供しない
- Kubernetesクラスタに既存のロギングソリューションを多数統合できる
Kubernetesでの基本的なログ記録
1秒に1回テキストを標準出力するポッド
debug/counter-pod.yaml
apiVersion: v1 kind: Pod metadata: name: counter spec: containers: - name: count image: busybox args: [/bin/sh, -c, 'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done']
$ kubectl apply -f debug/counter-pod.yaml pod/counter created $ kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES counter 1/1 Running 0 14s 10.244.1.84 kb2 <none> <none> $ kubectl describe pod counter Name: counter Namespace: default Priority: 0 PriorityClassName: <none> Node: kb2/192.168.0.102 Start Time: Mon, 11 Mar 2019 19:44:53 +0900 Labels: <none> Annotations: cni.projectcalico.org/podIP: 10.244.1.84/32 kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"name":"counter","namespace":"default"},"spec":{"containers":[{"args":["/bin/... Status: Running IP: 10.244.1.84 Containers: count: Container ID: docker://5e5c7d509ce1bfadac97749fb855f5771042fce001972efb7b1874cf769fbb2f Image: busybox Image ID: docker-pullable://busybox@sha256:061ca9704a714ee3e8b80523ec720c64f6209ad3f97c0ff7cb9ec7d19f15149f Port: <none> Host Port: <none> Args: /bin/sh -c i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done
pod:counter のログを確認
$ kubectl log counter log is DEPRECATED and will be removed in a future version. Use logs instead. 0: Mon Mar 11 10:44:59 UTC 2019 1: Mon Mar 11 10:45:00 UTC 2019 2: Mon Mar 11 10:45:01 UTC 2019 3: Mon Mar 11 10:45:02 UTC 2019 4: Mon Mar 11 10:45:03 UTC 2019 5: Mon Mar 11 10:45:04 UTC 2019 6: Mon Mar 11 10:45:05 UTC 2019 7: Mon Mar 11 10:45:06 UTC 2019 8: Mon Mar 11 10:45:07 UTC 2019 9: Mon Mar 11 10:45:08 UTC 2019 10: Mon Mar 11 10:45:09 UTC 2019 :
ノード上のログを確認してみる
kb2 の /var/log/containers を確認。counter_default_count-5e5c7d50... を追跡する。/var/log/pods/b460364c-43ea-11e9-8f5d-9ca3ba319985/count/0.log. にリンクしている
$ ls -al /var/log/containers total 80 drwxr-xr-x. 2 root root 12288 Mar 11 19:44 . drwxr-xr-x. 7 root root 4096 Jan 25 20:38 .. lrwxrwxrwx. 1 root root 68 Feb 28 23:58 calico-node-jdbbb_kube-system_calico-node-44f1f3e3277039d626404d7f4f4f9bdb97349b762ba38ce6e4fac50cf21a1c2b.log -> /var/log/pods/a206ccdc-2f85-11e9-83f4-9ca3ba319985/calico-node/3.log lrwxrwxrwx. 1 root root 68 Feb 24 16:03 calico-node-jdbbb_kube-system_calico-node-80765d945664af23866537c55b7dcfead8a0ce3e8e9649494621f35104fd44c9.log -> /var/log/pods/a206ccdc-2f85-11e9-83f4-9ca3ba319985/calico-node/2.log lrwxrwxrwx. 1 root root 68 Feb 24 16:03 calico-node-jdbbb_kube-system_install-cni-3c4e153bae5aa417322473e704275c3ec5a2e541a4fdafa022e2d92009183bfe.log -> /var/log/pods/a206ccdc-2f85-11e9-83f4-9ca3ba319985/install-cni/2.log lrwxrwxrwx. 1 root root 68 Feb 28 23:58 calico-node-jdbbb_kube-system_install-cni-4799ede6da39807478b6316ce18957d255575132d620d524a06b6e98566e8948.log -> /var/log/pods/a206ccdc-2f85-11e9-83f4-9ca3ba319985/install-cni/3.log lrwxrwxrwx. 1 root root 62 Mar 11 19:44 counter_default_count-5e5c7d509ce1bfadac97749fb855f5771042fce001972efb7b1874cf769fbb2f.log -> /var/log/pods/b460364c-43ea-11e9-8f5d-9ca3ba319985/count/0.log lrwxrwxrwx. 1 root root 67 Feb 28 23:58 kube-proxy-x6gbc_kube-system_kube-proxy-2acc29cfc69b74874faa11d4c9a793a7a43b9925575c112a4015f7af39ee09c8.log -> /var/log/pods/a204dab0-2f85-11e9-83f4-9ca3ba319985/kube-proxy/4.log lrwxrwxrwx. 1 root root 67 Feb 24 16:03 kube-proxy-x6gbc_kube-system_kube-proxy-942c1bd44c48442bfe17c9add49cbad796ac224f00fb89b2e2eabe834739de05.log -> /var/log/pods/a204dab0-2f85-11e9-83f4-9ca3ba319985/kube-proxy/3.log
/var/log/pods には b460364c-43ea-11e9-8f5d-9ca3ba319985 があって...
$ ls -al /var/log/pods total 40 drwxr-xr-x. 5 root root 4096 Mar 11 19:44 . drwxr-xr-x. 7 root root 4096 Jan 25 20:38 .. drwxr-xr-x. 3 root root 4096 Feb 13 20:51 a204dab0-2f85-11e9-83f4-9ca3ba319985 drwxr-xr-x. 4 root root 4096 Feb 13 20:51 a206ccdc-2f85-11e9-83f4-9ca3ba319985 drwxr-xr-x. 3 root root 4096 Mar 11 19:44 b460364c-43ea-11e9-8f5d-9ca3ba319985
/var/log/pods/b460364c-.../ には count があって...
$ ls -al /var/log/pods/b460364c-43ea-11e9-8f5d-9ca3ba319985 total 24 drwxr-xr-x. 3 root root 4096 Mar 11 19:44 . drwxr-xr-x. 5 root root 4096 Mar 11 19:44 .. drwxr-xr-x. 2 root root 4096 Mar 11 19:44 count
/var/log/pods/b460364c-.../count/ には 0.log があって /var/lib/docker/containers/5e5c7d5.../5e5c7d5...-json.log にリンクしている
$ ls -al /var/log/pods/b460364c-43ea-11e9-8f5d-9ca3ba319985/count total 24 drwxr-xr-x. 2 root root 4096 Mar 11 19:44 . drwxr-xr-x. 3 root root 4096 Mar 11 19:44 .. lrwxrwxrwx. 1 root root 165 Mar 11 19:44 0.log -> /var/lib/docker/containers/5e5c7d509ce1bfadac97749fb855f5771042fce001972efb7b1874cf769fbb2f/5e5c7d509ce1bfadac97749fb855f5771042fce001972efb7b1874cf769fbb2f-json.log
/var/lib/docker/containers/5e5c7d5.../5e5c7d5...-json.log の内容を確認。json で log フィールドに記録されていた。
$ sudo tail /var/lib/docker/containers/5e5c7d509ce1bfadac97749fb855f5771042fce001972efb7b1874cf769fbb2f/5e5c7d509ce1bfadac97749fb855f5771042fce001972efb7b1874cf769fbb2f-json.log {"log":"707: Mon Mar 11 10:56:47 UTC 2019\n","stream":"stdout","time":"2019-03-11T10:56:47.219187708Z"} {"log":"708: Mon Mar 11 10:56:48 UTC 2019\n","stream":"stdout","time":"2019-03-11T10:56:48.220973929Z"} {"log":"709: Mon Mar 11 10:56:49 UTC 2019\n","stream":"stdout","time":"2019-03-11T10:56:49.222537805Z"} {"log":"710: Mon Mar 11 10:56:50 UTC 2019\n","stream":"stdout","time":"2019-03-11T10:56:50.223903951Z"} {"log":"711: Mon Mar 11 10:56:51 UTC 2019\n","stream":"stdout","time":"2019-03-11T10:56:51.225434989Z"} {"log":"712: Mon Mar 11 10:56:52 UTC 2019\n","stream":"stdout","time":"2019-03-11T10:56:52.226873482Z"} {"log":"713: Mon Mar 11 10:56:53 UTC 2019\n","stream":"stdout","time":"2019-03-11T10:56:53.22864989Z"} {"log":"714: Mon Mar 11 10:56:54 UTC 2019\n","stream":"stdout","time":"2019-03-11T10:56:54.230080744Z"} {"log":"715: Mon Mar 11 10:56:55 UTC 2019\n","stream":"stdout","time":"2019-03-11T10:56:55.231569155Z"} {"log":"716: Mon Mar 11 10:56:56 UTC 2019\n","stream":"stdout","time":"2019-03-11T10:56:56.233325322Z"}