tjtjtjのメモ

自分のためのメモです。誰かの役には立ちません。

kubernetes 学習 taints tolerations その2

pod でなく deployment で taints/tolerations を試す。

確認

$ kubectl describe node kb1
Taints:             node-role.kubernetes.io/master:NoSchedule

$ kubectl describe node kb2
Taints:             <none>

$ kubectl describe node kb3
Taints:             <none>

テキトーな deployment.yaml

nginx-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:                ## <--- PodSpec
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80
      tolerations:
      - key: apptype
        value: appproxy
        effect: NoExecute

nginx-deployment.yaml を適用

$ kubectl apply -f nginx-deployment.yaml
deployment.apps/nginx-deployment created

pod確認 kb2,kb3 で実行されている。

$ kubectl get pod -o wide
NAME                                READY   STATUS    RESTARTS   AGE   IP            NODE   NOMINATED NODE   READINESS GATES
nginx-deployment-78d48847c6-5ldbv   1/1     Running   0          11m   10.244.2.33   kb3    <none>           <none>
nginx-deployment-78d48847c6-wh7x9   1/1     Running   0          11m   10.244.1.41   kb2    <none>           <none>

$ kubectl describe pod | grep -3 Tole
Tolerations:     apptype=appproxy:NoExecute
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s

Tolerations:     apptype=appproxy:NoExecute
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s

kb2 から追い出して kb3で実行する

kb2 を汚す。

$ kubectl taint node kb2 apptype=asdf:NoExecute
node/kb2 tainted

$ kubectl get pod -o wide
NAME                                READY   STATUS    RESTARTS   AGE   IP            NODE   NOMINATED NODE   READINESS GATES
nginx-deployment-78d48847c6-5ldbv   1/1     Running   0          12m   10.244.2.33   kb3    <none>           <none>
nginx-deployment-78d48847c6-j5gw5   1/1     Running   0          16s   10.244.2.34   kb3    <none>           <none>

kb3 から追い出して kb2 で実行する

kb2 の汚れを取る。

$ kubectl taint node kb2 apptype-
node/kb2 untainted

追い出し前の確認

$ kubectl get pod -o wide
NAME                                READY   STATUS    RESTARTS   AGE   IP            NODE   NOMINATED NODE   READINESS GATES
nginx-deployment-78d48847c6-5ldbv   1/1     Running   0          13m   10.244.2.33   kb3    <none>           <none>
nginx-deployment-78d48847c6-j5gw5   1/1     Running   0          91s   10.244.2.34   kb3    <none>           <none>

kb3を汚す。kb2からkb3に引っ越している。

$ kubectl taint node kb3 apptype=asdf:NoExecute
node/kb3 tainted

$ kubectl get pod -o wide
NAME                                READY   STATUS              RESTARTS   AGE     IP            NODE   NOMINATED NODE   READINESS GATES
nginx-deployment-78d48847c6-5ldbv   1/1     Terminating         0          14m     10.244.2.33   kb3    <none>           <none>
nginx-deployment-78d48847c6-cxdpj   0/1     ContainerCreating   0          2s      <none>        kb2    <none>           <none>
nginx-deployment-78d48847c6-j5gw5   1/1     Terminating         0          2m24s   10.244.2.34   kb3    <none>           <none>
nginx-deployment-78d48847c6-v8nkt   1/1     Running             0          2s      10.244.1.43   kb2    <none>           <none>
$ kubectl get pod -o wide
NAME                                READY   STATUS        RESTARTS   AGE     IP            NODE   NOMINATED NODE   READINESS GATES
nginx-deployment-78d48847c6-5ldbv   0/1     Terminating   0          14m     <none>        kb3    <none>           <none>
nginx-deployment-78d48847c6-cxdpj   1/1     Running       0          5s      10.244.1.42   kb2    <none>           <none>
nginx-deployment-78d48847c6-j5gw5   0/1     Terminating   0          2m27s   10.244.2.34   kb3    <none>           <none>
nginx-deployment-78d48847c6-v8nkt   1/1     Running       0          5s      10.244.1.43   kb2    <none>           <none>
$ kubectl get pod -o wide
NAME                                READY   STATUS    RESTARTS   AGE   IP            NODE   NOMINATED NODE   READINESS GATES
nginx-deployment-78d48847c6-cxdpj   1/1     Running   0          15s   10.244.1.42   kb2    <none>           <none>
nginx-deployment-78d48847c6-v8nkt   1/1     Running   0          15s   10.244.1.43   kb2    <none>           <none>

おわり

pod は 追い出すと delete されて終了だが、deployment の pod 死んでも生き延びようとする?書籍にあたったほうがよさそう。

kubernetes 学習 taints tolerations

coredns が起動しないのを taints tolerations のせいかと勘違いしちょっといじったが違った。今回はちゃんと taints tolerations 試す。

ありがたい資料

Taints and Tolerations https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/

ポッドが不適切なノードにスケジュールされないようにする。 ノードが汚染を容認しないポッドを受け入れない。

Kubernetesのtaintsとtolerationsについて https://qiita.com/sheepland/items/8fedae15e157c102757f

最初の確認

$ kubectl describe node kb1
Name:               kb1
Roles:              master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/hostname=kb1
                    node-role.kubernetes.io/master=
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.
                    node.alpha.kubernetes.io/ttl: 0
                    projectcalico.org/IPv4Address: 192.168.0.101/24
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Wed, 13 Feb 2019 20:28:37 +0900
Taints:             node-role.kubernetes.io/master:NoSchedule
Unschedulable:      false
$ kubectl describe node kb2
Name:               kb2
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/hostname=kb2
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
                    node.alpha.kubernetes.io/ttl: 0
                    projectcalico.org/IPv4Address: 192.168.0.102/24
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Wed, 13 Feb 2019 20:51:00 +0900
Taints:             <none>
Unschedulable:      false
$ kubectl describe node kb3
Name:               kb3
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/hostname=kb3
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
                    node.alpha.kubernetes.io/ttl: 0
                    projectcalico.org/IPv4Address: 192.168.0.103/24
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Wed, 13 Feb 2019 20:51:51 +0900
Taints:             <none>
Unschedulable:      false

taint, toleration なし

pod-nginx.yaml

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx
  replicas: 1

てきとーなpod適用。kb2で動作中。

$ kubectl create -f pod-nginx.yaml
pod/nginx created
$ kubectl get pod -o wide
NAME    READY   STATUS    RESTARTS   AGE   IP            NODE   NOMINATED NODE   READINESS GATES
nginx   1/1     Running   0          55s   10.244.1.39   kb2    <none>           <none>

$ kubectl delete pod nginx pod "nginx" deleted $ kubectl get pod -o wide No resources found.

taint, toleration 設定 許容nodeあり

node:kb2 に taint:key1=test を設定。 node:kb3 に taint:key1=prod を設定。 kb2 はテスト用、kb3 は本番用みたいな感じ。

$ kubectl taint node kb2 key1=test:NoSchedule
node/kb2 tainted
$ kubectl describe node kb2
:
Taints:             key1=test:NoSchedule
:
$ kubectl taint node kb3 key1=prod:NoSchedule
node/kb3 tainted
$ kubectl describe node kb3
:
Taints:             key1=prod:NoSchedule
:

pod:nginx は実は本番用だったってことにする。 toleration:prod:NoSchedule を設定。

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx
  replicas: 1
  tolerations:
  - key: key1
    value: prod
    effect: NoSchedule

node:kb3 で pod:nginx がrunning

$ kubectl create -f pod-nginx.yaml
pod/nginx created
$ kubectl describe pod nginx 
:
Tolerations:     key1=prod:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
:
$ kubectl get pod -o wide
NAME    READY   STATUS    RESTARTS   AGE   IP            NODE   NOMINATED NODE   READINESS GATES
nginx   1/1     Running   0          13s   10.244.2.30   kb3    <none>           <none>

許容nodeをなくし追い出す。

node:kb3 の taint 変えてみる。kb3 から追い出された。

$ kubectl taint node kb3 key1=test:NoExecute
node/kb3 tainted
$ kubectl get pod -o wide
NAME    READY   STATUS        RESTARTS   AGE     IP            NODE   NOMINATED NODE   READINESS GATES
nginx   0/1     Terminating   0          4m43s   10.244.2.31   kb3    <none>           <none>
$ kubectl get pod -o wide
No resources found.
$ kubectl describe pod nginx
Error from server (NotFound): pods "nginx" not found

お掃除

$ kubectl taint node kb2 key1-
$ kubectl taint node kb3 key1-

うーん

pod でなくdeployment で試すんだった。テスト/本番が同じクラスタってのもオカシイ。

kubernetes 学習 クラスタ作り直し

knative 試してたらグッチャグッチャになったので、クラスタを再構築したメモ。

マスターでワーカーノードを削除

$ kubectl get node
NAME   STATUS     ROLES    AGE   VERSION
kb1    Ready      master   14d   v1.13.2
kb2    Ready      <none>   13d   v1.13.2
kb3    Ready      <none>   13d   v1.13.2

$ kubectl delete node kb2
node "kb2" deleted

$ kubectl delete node kb3
node "kb3" deleted

$ kubectl get node
NAME   STATUS   ROLES    AGE   VERSION
kb1    Ready    master   14d   v1.13.2

ワーカーからkubeadm reset

$ sudo kubeadm reset

[reset] WARNING: changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] are you sure you want to proceed? [y/N]: y
[preflight] running pre-flight checks
[reset] no etcd config found. Assuming external etcd
[reset] please manually reset etcd to prevent further issues
[reset] stopping the kubelet service
[reset] unmounting mounted directories in "/var/lib/kubelet"
[reset] deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]
[reset] deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually.
For example:
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

マスターでも kubeadm reset

$ sudo kubeadm reset
[reset] WARNING: changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] are you sure you want to proceed? [y/N]: y
[preflight] running pre-flight checks
[reset] Reading configuration from the cluster...
[reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oya ml'
[reset] stopping the kubelet service
[reset] unmounting mounted directories in "/var/lib/kubelet"
[reset] deleting contents of stateful directories: [/var/lib/etcd /var/lib/kubelet /etc/cni/net.d /va r/lib/dockershim /var/run/kubernetes]
[reset] deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/boot strap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually.
For example:
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

マスターで kubeadm init

ここからはいつもの流れ。join のtokenをメモ

$ sudo kubeadm init --pod-network-cidr=10.244.0.0/16
:
:
  kubeadm join 192.168.0.101:6443 --token nkm1pm.4wmal1abbb9j366m --discovery-token-ca-cert-hash sha256:658c4ff9233cab17542af787a0ae02002a8584f4badbcd2d30742ad8b59825ed
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown core:core $HOME/.kube/config
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.2", GitCommit:"cff46ab41ff0bb44d8584413b598ad8360ec1def", GitTreeState:"clean", BuildDate:"2019-01-10T23:35:51Z", GoVersion:"go1.11.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:00:57Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

$ kubectl get nodes
NAME   STATUS     ROLES    AGE   VERSION
kb1    NotReady   master   61s   v1.13.2

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                          READY   STATUS    RESTARTS   AGE
kube-system   coredns-86c58d9df4-6n9c8      0/1     Pending   0          76s
kube-system   coredns-86c58d9df4-jbldk      0/1     Pending   0          76s
kube-system   etcd-kb1                      1/1     Running   0          24s
kube-system   kube-apiserver-kb1            1/1     Running   0          16s
kube-system   kube-controller-manager-kb1   1/1     Running   0          35s
kube-system   kube-proxy-44cb5              1/1     Running   0          76s
kube-system   kube-scheduler-kb1            1/1     Running   0          21s

coredns が Pending のまま動かない。

「kubectl describe pod coredns-...」 してエラー「0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.」 なので、 Tolerations/Taints を調べ、pod がヨゴレ仕事(taint)で node がヨゴレ役を引き受ける(toleration)ってイメージと逆で悩んだり、 CriticalAddonsOnly を taint するのかとか試したりもした。

Kubernetesのtaintsとtolerationsについて https://qiita.com/sheepland/items/8fedae15e157c102757f

が、そういうことでなく、こういうことだった。

「これは予想通りで、設計の一部です。 kubeadmはネットワークプロバイダに依存しないため、管理者は選択したPodネットワークソリューションをインストールする必要があります。 CoreDNSを完全に展開する前に、Podネットワークをインストールする必要があります。 したがって、ネットワークがセットアップされる前のPending状態です。」 https://kubernetes.io/docs/setup/independent/troubleshooting-kubeadm/#coredns-or-kube-dns-is-stuck-in-the-pending-state

つまりこれやってなかったのだ。

$ kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
$ kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

running になりました。

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                          READY   STATUS    RESTARTS   AGE
kube-system   calico-node-hdjxz             2/2     Running   0          23s
kube-system   coredns-86c58d9df4-6n9c8      1/1     Running   0          5m39s
kube-system   coredns-86c58d9df4-jbldk      1/1     Running   0          5m39s
kube-system   etcd-kb1                      1/1     Running   0          4m47s
kube-system   kube-apiserver-kb1            1/1     Running   0          4m39s
kube-system   kube-controller-manager-kb1   1/1     Running   0          4m58s
kube-system   kube-proxy-44cb5              1/1     Running   0          5m39s
kube-system   kube-scheduler-kb1            1/1     Running   0          4m44s

ワーカーで kubeadm join

$ sudo kubeadm join 192.168.0.101:6443 --token nkm1pm.4wmal1abbb9j366m --discovery-token-ca-cert-hash sha256:658c4ff9233cab17542af787a0ae02002a8584f4badbcd2d30742ad8b59825ed

マスターでクラスタ確認

$ kubectl get node
NAME   STATUS     ROLES    AGE   VERSION
kb1    Ready      master   23m   v1.13.2
kb2    Ready      <none>   55s   v1.13.2
kb3    NotReady   <none>   4s    v1.13.2

Kubernetes 学習 Performing a Rolling Update を自前クラスタで復習

Performing a Rolling Update

https://kubernetes.io/docs/tutorials/kubernetes-basics/update/update-intro/

前回の続きを確認

$ kubectl get nodes
NAME   STATUS   ROLES    AGE   VERSION
kb1    Ready    master   10d   v1.13.2
kb2    Ready    <none>   10d   v1.13.2
kb3    Ready    <none>   10d   v1.13.2

$ kubectl get deployments
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
kubernetes-bootcamp   2/2     2            2           18m

$ kubectl get pods -o wide
NAME                                   READY   STATUS    RESTARTS   AGE   IP            NODE   NOMINATED NODE   READINESS GATES
kubernetes-bootcamp-6bf84cb898-6k94z   1/1     Running   0          16m   10.244.2.10   kb3    <none>           <none>
kubernetes-bootcamp-6bf84cb898-qswcf   1/1     Running   0          18m   10.244.2.9    kb3    <none>           <none>

$ kubectl get services
NAME               TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
bootcamp-service   LoadBalancer   10.108.87.251   <pending>     8080:31729/TCP   10m
kubernetes         ClusterIP      10.96.0.1       <none>        443/TCP          10d

チュートリアルは replicas:4 でスタートするので スケールさせる

$ kubectl scale deployments/kubernetes-bootcamp --replicas=4
deployment.extensions/kubernetes-bootcamp scaled

$ kubectl get pods -o wide
NAME                                   READY   STATUS    RESTARTS   AGE   IP            NODE   NOMINATED NODE   READINESS GATES
kubernetes-bootcamp-6bf84cb898-6k94z   1/1     Running   0          18m   10.244.2.10   kb3    <none>           <none>
kubernetes-bootcamp-6bf84cb898-gqms4   1/1     Running   0          4s    10.244.1.14   kb2    <none>           <none>
kubernetes-bootcamp-6bf84cb898-hnl46   1/1     Running   0          4s    10.244.1.13   kb2    <none>           <none>
kubernetes-bootcamp-6bf84cb898-qswcf   1/1     Running   0          21m   10.244.2.9    kb3    <none>           <none>

Step 1: Update the version of the app

確認

$ kubectl get deployments
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
kubernetes-bootcamp   4/4     4            4           22m

$ kubectl get pods
NAME                                   READY   STATUS    RESTARTS   AGE
kubernetes-bootcamp-6bf84cb898-6k94z   1/1     Running   0          20m
kubernetes-bootcamp-6bf84cb898-gqms4   1/1     Running   0          79s
kubernetes-bootcamp-6bf84cb898-hnl46   1/1     Running   0          79s
kubernetes-bootcamp-6bf84cb898-qswcf   1/1     Running   0          22m

pods 詳細確認。長いので2コ目以降は省略

$ kubectl describe pods
Name:               kubernetes-bootcamp-6bf84cb898-6k94z
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               kb3/192.168.0.103
Start Time:         Tue, 05 Feb 2019 20:05:30 +0900
Labels:             pod-template-hash=6bf84cb898
                    run=kubernetes-bootcamp
Annotations:        cni.projectcalico.org/podIP: 10.244.2.10/32
Status:             Running
IP:                 10.244.2.10
Controlled By:      ReplicaSet/kubernetes-bootcamp-6bf84cb898
Containers:
  kubernetes-bootcamp:
    Container ID:   docker://81839f7a06215d94260e38103f735cd51625b555ec8242d24eecc5bf52ad6d05
    Image:          gcr.io/google-samples/kubernetes-bootcamp:v1
    Image ID:       docker-pullable://gcr.io/google-samples/kubernetes-bootcamp@sha256:0d6b8ee63bb57c5f5b6156f446b3bc3b3c143d233037f3a2f00e279c8fcc64af
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Tue, 05 Feb 2019 20:05:32 +0900
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-88mmv (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  default-token-88mmv:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-88mmv
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  20m   default-scheduler  Successfully assigned default/kubernetes-bootcamp-6bf84cb898-6k94z to kb3
  Normal  Pulled     20m   kubelet, kb3       Container image "gcr.io/google-samples/kubernetes-bootcamp:v1" already present on machine
  Normal  Created    20m   kubelet, kb3       Created container
  Normal  Started    20m   kubelet, kb3       Started container

~略~

kubernetes-bootcamp のイメージを jocatalin/kubernetes-bootcamp:v2 にする。

$ kubectl set image deployments/kubernetes-bootcamp kubernetes-bootcamp=jocatalin/kubernetes-bootcamp:v2
deployment.extensions/kubernetes-bootcamp image updated

Step 2: Verify an update

イメージ更新を確認。v1 のpod がterminateして、v2のpodがrun してきた。

$ kubectl get pods
NAME                                   READY   STATUS        RESTARTS   AGE
kubernetes-bootcamp-5bf4d5689b-4gd7q   1/1     Running       0          36s
kubernetes-bootcamp-5bf4d5689b-4xzxq   1/1     Running       0          31s
kubernetes-bootcamp-5bf4d5689b-dsh7s   1/1     Running       0          36s
kubernetes-bootcamp-5bf4d5689b-kpbvc   1/1     Running       0          31s
kubernetes-bootcamp-6bf84cb898-6k94z   1/1     Terminating   0          22m
kubernetes-bootcamp-6bf84cb898-gqms4   1/1     Terminating   0          3m40s
kubernetes-bootcamp-6bf84cb898-hnl46   0/1     Terminating   0          3m40s
kubernetes-bootcamp-6bf84cb898-qswcf   1/1     Terminating   0          25m

$ kubectl get pods
NAME                                   READY   STATUS    RESTARTS   AGE
kubernetes-bootcamp-5bf4d5689b-4gd7q   1/1     Running   0          5m47s
kubernetes-bootcamp-5bf4d5689b-4xzxq   1/1     Running   0          5m42s
kubernetes-bootcamp-5bf4d5689b-dsh7s   1/1     Running   0          5m47s
kubernetes-bootcamp-5bf4d5689b-kpbvc   1/1     Running   0          5m42s

deployments/kubernetes-bootcamp の詳細確認 image: jocatalin/kubernetes-bootcamp:v2 になっている。

$ kubectl describe deployments/kubernetes-bootcamp
Name:                   kubernetes-bootcamp
Namespace:              default
CreationTimestamp:      Tue, 05 Feb 2019 20:02:55 +0900
Labels:                 run=kubernetes-bootcamp
Annotations:            deployment.kubernetes.io/revision: 2
Selector:               run=kubernetes-bootcamp
Replicas:               4 desired | 4 updated | 4 total | 4 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  run=kubernetes-bootcamp
  Containers:
   kubernetes-bootcamp:
    Image:        jocatalin/kubernetes-bootcamp:v2
    Port:         8080/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   kubernetes-bootcamp-5bf4d5689b (4/4 replicas created)
Events:
  Type    Reason             Age                  From                   Message
  ----    ------             ----                 ----                   -------
  Normal  ScalingReplicaSet  28m                  deployment-controller  Scaled up replica set kubernetes-bootcamp-6bf84cb898 to 1
  Normal  ScalingReplicaSet  6m51s (x2 over 25m)  deployment-controller  Scaled up replica set kubernetes-bootcamp-6bf84cb898 to 4
  Normal  ScalingReplicaSet  3m47s                deployment-controller  Scaled up replica set kubernetes-bootcamp-5bf4d5689b to 1
  Normal  ScalingReplicaSet  3m47s                deployment-controller  Scaled down replica set kubernetes-bootcamp-6bf84cb898 to 3
  Normal  ScalingReplicaSet  3m47s                deployment-controller  Scaled up replica set kubernetes-bootcamp-5bf4d5689b to 2
  Normal  ScalingReplicaSet  3m42s (x2 over 15m)  deployment-controller  Scaled down replica set kubernetes-bootcamp-6bf84cb898 to 2
  Normal  ScalingReplicaSet  3m42s                deployment-controller  Scaled up replica set kubernetes-bootcamp-5bf4d5689b to 3
  Normal  ScalingReplicaSet  3m42s                deployment-controller  Scaled down replica set kubernetes-bootcamp-6bf84cb898 to 1
  Normal  ScalingReplicaSet  3m42s                deployment-controller  Scaled up replica set kubernetes-bootcamp-5bf4d5689b to 4
  Normal  ScalingReplicaSet  3m40s                deployment-controller  Scaled down replica set kubernetes-bootcamp-6bf84cb898 to 0

pods の詳細確認。ながいので略

$ kubectl describe pods
Name:               kubernetes-bootcamp-5bf4d5689b-4gd7q
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               kb2/192.168.0.102
Start Time:         Tue, 05 Feb 2019 20:27:29 +0900
Labels:             pod-template-hash=5bf4d5689b
                    run=kubernetes-bootcamp
Annotations:        cni.projectcalico.org/podIP: 10.244.1.15/32
Status:             Running
IP:                 10.244.1.15
Controlled By:      ReplicaSet/kubernetes-bootcamp-5bf4d5689b
Containers:
  kubernetes-bootcamp:
    Container ID:   docker://1912a6a10c45f44ef1267ed7d585abed509fdc27ffcfcd8c8b85d22e087d87b6
    Image:          jocatalin/kubernetes-bootcamp:v2
    Image ID:       docker-pullable://jocatalin/kubernetes-bootcamp@sha256:fb1a3ced00cecfc1f83f18ab5cd14199e30adc1b49aa4244f5d65ad3f5feb2a5
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Tue, 05 Feb 2019 20:27:33 +0900
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-88mmv (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  default-token-88mmv:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-88mmv
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  5m1s   default-scheduler  Successfully assigned default/kubernetes-bootcamp-5bf4d5689b-4gd7q to kb2
  Normal  Pulling    5m     kubelet, kb2       pulling image "jocatalin/kubernetes-bootcamp:v2"
  Normal  Pulled     4m57s  kubelet, kb2       Successfully pulled image "jocatalin/kubernetes-bootcamp:v2"
  Normal  Created    4m57s  kubelet, kb2       Created container
  Normal  Started    4m57s  kubelet, kb2       Started container


~略~

curl 連打。各ポッドがお仕事しています。

$ curl http://10.108.87.251:8080
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-5bf4d5689b-dsh7s | v=2

$ curl http://10.108.87.251:8080
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-5bf4d5689b-4gd7q | v=2

$ curl http://10.108.87.251:8080
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-5bf4d5689b-kpbvc | v=2

$ curl http://10.108.87.251:8080
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-5bf4d5689b-4xzxq | v=2

ロールアウトステータスコマンドを実行しても、アップデートを確認できるそうだ。 successfully rolled out

$ kubectl rollout status deployments/kubernetes-bootcamp
deployment "kubernetes-bootcamp" successfully rolled out

pods の詳細は略

Step 3: Rollback an update

チュートリアルやったときここ意味が分からなかった。

もう一度アップデートを実行して、v10としてタグ付けされたイメージをデプロイしましょう。

だそうです。

$ kubectl set image deployments/kubernetes-bootcamp kubernetes-bootcamp=gcr.io/google-samples/kubernetes-bootcamp:v10
deployment.extensions/kubernetes-bootcamp image updated

status:ImagePullBackOff <-> ErrImagePull を繰り返す。これは v10 のイメージがないから。

$ kubectl get pods
NAME                                   READY   STATUS             RESTARTS   AGE
kubernetes-bootcamp-597cfc5b76-flxjt   0/1     ImagePullBackOff   0          28s
kubernetes-bootcamp-597cfc5b76-zrshs   0/1     ImagePullBackOff   0          28s
kubernetes-bootcamp-5bf4d5689b-4gd7q   1/1     Running            0          17m
kubernetes-bootcamp-5bf4d5689b-4xzxq   1/1     Running            0          17m
kubernetes-bootcamp-5bf4d5689b-dsh7s   1/1     Running            0          17m
kubernetes-bootcamp-5bf4d5689b-kpbvc   1/1     Terminating        0          17m

$ kubectl get deployments
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
kubernetes-bootcamp   3/4     2            3           42m

@kb1 ~ $ kubectl get pods
NAME                                   READY   STATUS         RESTARTS   AGE
kubernetes-bootcamp-597cfc5b76-flxjt   0/1     ErrImagePull   0          39s
kubernetes-bootcamp-597cfc5b76-zrshs   0/1     ErrImagePull   0          39s
kubernetes-bootcamp-5bf4d5689b-4gd7q   1/1     Running        0          17m
kubernetes-bootcamp-5bf4d5689b-4xzxq   1/1     Running        0          17m
kubernetes-bootcamp-5bf4d5689b-dsh7s   1/1     Running        0          17m

$ kubectl get pods
NAME                                   READY   STATUS             RESTARTS   AGE
kubernetes-bootcamp-597cfc5b76-flxjt   0/1     ImagePullBackOff   0          44s
kubernetes-bootcamp-597cfc5b76-zrshs   0/1     ImagePullBackOff   0          44s
kubernetes-bootcamp-5bf4d5689b-4gd7q   1/1     Running            0          17m
kubernetes-bootcamp-5bf4d5689b-4xzxq   1/1     Running            0          17m
kubernetes-bootcamp-5bf4d5689b-dsh7s   1/1     Running            0          17m

$ kubectl get pods
NAME                                   READY   STATUS         RESTARTS   AGE
kubernetes-bootcamp-597cfc5b76-flxjt   0/1     ErrImagePull   0          64s
kubernetes-bootcamp-597cfc5b76-zrshs   0/1     ErrImagePull   0          64s
kubernetes-bootcamp-5bf4d5689b-4gd7q   1/1     Running        0          17m
kubernetes-bootcamp-5bf4d5689b-4xzxq   1/1     Running        0          17m
kubernetes-bootcamp-5bf4d5689b-dsh7s   1/1     Running        0          17m

rollout を undo 。なるほど rollout がここで生きてくると。

$ kubectl rollout undo deployments/kubernetes-bootcamp
deployment.extensions/kubernetes-bootcamp rolled back
$ kubectl get pods
NAME                                   READY   STATUS    RESTARTS   AGE
kubernetes-bootcamp-5bf4d5689b-4gd7q   1/1     Running   0          21m
kubernetes-bootcamp-5bf4d5689b-4xzxq   1/1     Running   0          21m
kubernetes-bootcamp-5bf4d5689b-dsh7s   1/1     Running   0          21m
kubernetes-bootcamp-5bf4d5689b-rtxj8   1/1     Running   0          5s

お掃除

$ kubectl delete services bootcamp-service
service "bootcamp-service" deleted

$ kubectl get services
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   10d

$ kubectl delete deployment kubernetes-bootcamp
deployment.extensions "kubernetes-bootcamp" deleted

$ kubectl get deployment 
No resources found.

$ kubectl get pods
NAME                                   READY   STATUS        RESTARTS   AGE
kubernetes-bootcamp-5bf4d5689b-4gd7q   1/1     Terminating   0          32m
kubernetes-bootcamp-5bf4d5689b-4xzxq   1/1     Terminating   0          32m
kubernetes-bootcamp-5bf4d5689b-dsh7s   1/1     Terminating   0          32m
kubernetes-bootcamp-5bf4d5689b-rtxj8   1/1     Terminating   0          10m

$ kubectl get pods
No resources found.

Kubernetes 学習 Scale Your App を自前クラスタで復習

Scale Your App

https://kubernetes.io/docs/tutorials/kubernetes-basics/scale/scale-intro/

チュートリアルの準備

$ kubectl get nodes
NAME   STATUS   ROLES    AGE   VERSION
kb1    Ready    master   10d   v1.13.2
kb2    Ready    <none>   10d   v1.13.2
kb3    Ready    <none>   10d   v1.13.2

$ kubectl get deployments
No resources found.

$ kubectl get pods
No resources found.

$ kubectl get services
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   10d

kubernetes-bootcamp をデプロイ

チュートリアルで使う kubernetes-bootcamp をデプロイ。podは1コ。

$ kubectl run kubernetes-bootcamp --image=gcr.io/google-samples/kubernetes-bootcamp:v1 --port=8080
deployment "kubernetes-bootcamp" created

$ kubectl get deployments
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
kubernetes-bootcamp   0/1     1            0           8s

$ kubectl get deployments
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
kubernetes-bootcamp   1/1     1            1           20s

$ kubectl get pods
NAME                                   READY   STATUS    RESTARTS   AGE
kubernetes-bootcamp-6bf84cb898-qswcf   1/1     Running   0          2m4s

Step 1: Scaling a deployment

スケールアップ

$ kubectl scale deployments/kubernetes-bootcamp --replicas=4
deployment.extensions/kubernetes-bootcamp scaled

スケールアップを確認。オプション -o wide で pod:node を確認できる

$ kubectl get deployments
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
kubernetes-bootcamp   2/4     4            2           2m47s

$ kubectl get deployments
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
kubernetes-bootcamp   4/4     4            4           2m51s

$ kubectl get pods
NAME                                   READY   STATUS    RESTARTS   AGE
kubernetes-bootcamp-6bf84cb898-6k94z   1/1     Running   0          35s
kubernetes-bootcamp-6bf84cb898-jz9wf   1/1     Running   0          35s
kubernetes-bootcamp-6bf84cb898-qswcf   1/1     Running   0          3m10s
kubernetes-bootcamp-6bf84cb898-vlwg6   1/1     Running   0          35s

$ kubectl get pods -o wide
NAME                                   READY   STATUS    RESTARTS   AGE     IP            NODE   NOMINATED NODE   READINESS GATES
kubernetes-bootcamp-6bf84cb898-6k94z   1/1     Running   0          79s     10.244.2.10   kb3    <none>           <none>
kubernetes-bootcamp-6bf84cb898-jz9wf   1/1     Running   0          79s     10.244.1.11   kb2    <none>           <none>
kubernetes-bootcamp-6bf84cb898-qswcf   1/1     Running   0          3m54s   10.244.2.9    kb3    <none>           <none>
kubernetes-bootcamp-6bf84cb898-vlwg6   1/1     Running   0          79s     10.244.1.12   kb2    <none>           <none>

deployments/kubernetes-bootcamp 詳細確認

$ kubectl describe deployments/kubernetes-bootcamp
Name:                   kubernetes-bootcamp
Namespace:              default
CreationTimestamp:      Tue, 05 Feb 2019 20:02:55 +0900
Labels:                 run=kubernetes-bootcamp
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               run=kubernetes-bootcamp
Replicas:               4 desired | 4 updated | 4 total | 4 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  run=kubernetes-bootcamp
  Containers:
   kubernetes-bootcamp:
    Image:        gcr.io/google-samples/kubernetes-bootcamp:v1
    Port:         8080/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   kubernetes-bootcamp-6bf84cb898 (4/4 replicas created)
Events:
  Type    Reason             Age    From                   Message
  ----    ------             ----   ----                   -------
  Normal  ScalingReplicaSet  4m41s  deployment-controller  Scaled up replica set kubernetes-bootcamp-6bf84cb898 to 1
  Normal  ScalingReplicaSet  2m6s   deployment-controller  Scaled up replica set kubernetes-bootcamp-6bf84cb898 to 4

Step 2: Load Balancing

LoadBalancer で expose。

$ kubectl expose deployment kubernetes-bootcamp --type=LoadBalancer --name=bootcamp-service
service/bootcamp-service exposed
$ kubectl get services bootcamp-service
NAME               TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
bootcamp-service   LoadBalancer   10.108.87.251   <pending>     8080:31729/TCP   44s
$ kubectl describe services bootcamp-service
Name:                     bootcamp-service
Namespace:                default
Labels:                   run=kubernetes-bootcamp
Annotations:              <none>
Selector:                 run=kubernetes-bootcamp
Type:                     LoadBalancer
IP:                       10.108.87.251
Port:                     <unset>  8080/TCP
TargetPort:               8080/TCP
NodePort:                 <unset>  31729/TCP
Endpoints:                10.244.1.11:8080,10.244.1.12:8080,10.244.2.10:8080 + 1 more...
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

curl 連打。各podが働いているのが分かる。

$ curl http://10.108.87.251:8080
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-6bf84cb898-qswcf | v=1

$ curl http://10.108.87.251:8080
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-6bf84cb898-vlwg6 | v=1

$ curl http://10.108.87.251:8080
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-6bf84cb898-6k94z | v=1

$ curl http://10.108.87.251:8080
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-6bf84cb898-jz9wf | v=1

Step 3: Scale Down

スケールダウン

$ kubectl scale deployments/kubernetes-bootcamp --replicas=2
deployment.extensions/kubernetes-bootcamp scaled

スケールダウンしたことを確認

$ kubectl get pods -o wide
NAME                                   READY   STATUS        RESTARTS   AGE   IP            NODE   NOMINATED NODE   READINESS GATES
kubernetes-bootcamp-6bf84cb898-6k94z   1/1     Running       0          10m   10.244.2.10   kb3    <none>           <none>
kubernetes-bootcamp-6bf84cb898-jz9wf   1/1     Terminating   0          10m   10.244.1.11   kb2    <none>           <none>
kubernetes-bootcamp-6bf84cb898-qswcf   1/1     Running       0          13m   10.244.2.9    kb3    <none>           <none>
kubernetes-bootcamp-6bf84cb898-vlwg6   1/1     Terminating   0          10m   10.244.1.12   kb2    <none>           <none>

$ kubectl get pods -o wide
NAME                                   READY   STATUS    RESTARTS   AGE   IP            NODE   NOMINATED NODE   READINESS GATES
kubernetes-bootcamp-6bf84cb898-6k94z   1/1     Running   0          12m   10.244.2.10   kb3    <none>           <none>
kubernetes-bootcamp-6bf84cb898-qswcf   1/1     Running   0          15m   10.244.2.9    kb3    <none>           <none>

curl 連打。スケールダウン後のpodがレスポンしたのが分かる。

$ curl http://10.108.87.251:8080
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-6bf84cb898-qswcf | v=1

$ curl http://10.108.87.251:8080
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-6bf84cb898-6k94z | v=1

Kubernetes 学習 Interactive Tutorial

Interactive Tutorial で投入した kubectl コマンドをメモ

Scale Your App

https://kubernetes.io/docs/tutorials/kubernetes-basics/scale/scale-intro/

$ kubectl get deployments
  deployments の確認

$ kubectl get pods
  pods の確認

$ kubectl scale deployments/kubernetes-bootcamp --replicas=4
  replica:4 にスケールアップ

$ kubectl get pods -o wide
  pods の確認。情報少し多め どこのnodeが分かる

$ kubectl describe deployments/kubernetes-bootcamp
  deployments/kubernetes-bootcamp の詳細を確認

$ kubectl scale deployments/kubernetes-bootcamp --replicas=2
  replica:4 にスケールダウン

Performing a Rolling Update

https://kubernetes.io/docs/tutorials/kubernetes-basics/update/update-intro/

$ kubectl get deployments

$ kubectl get pods

$ kubectl describe pods
  pods の詳細を確認

$ kubectl set image deployments/kubernetes-bootcamp kubernetes-bootcamp=jocatalin/kubernetes-bootcamp:v2
  deployments/kubernetes-bootcamp のイメージを jocatalin/kubernetes-bootcamp:v2 にする
  新イメージの podが起動し、旧イメージのpod が停止する

$ kubectl describe services/kubernetes-bootcamp
  サービスkubernetes-bootcamp の詳細確認

$ kubectl rollout status deployments/kubernetes-bootcamp
  ロールアウト状態を確認

$ kubectl set image deployments/kubernetes-bootcamp kubernetes-bootcamp=gcr.io/google-samples/kubernetes-bootcamp:v10
  deployments/kubernetes-bootcamp のイメージを gcr.io/google-samples/kubernetes-bootcamp:v10 にする
  そんなイメージはない
  新イメージの pod が起動しようとするが失敗、旧イメージの pod が動作を続ける

$ kubectl rollout undo deployments/kubernetes-bootcamp
  ロールアウトを取消

kubernetes 学習 nginx pod 作成

kb1 nginx pod 作成

$ kubectl get pods
No resources found.
$ kubectl run nginx --image=nginx:1.15.8
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
deployment.apps/nginx created
$ kubectl get pods
NAME                     READY   STATUS              RESTARTS   AGE
nginx-694b4b8479-swzm2   0/1     ContainerCreating   0          12s
$ kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
nginx-694b4b8479-swzm2   1/1     Running   0          21s
$ kubectl get deployment
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   1/1     1            1           2m40s

ロードバランサ 作成

$ kubectl get service
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP      10.96.0.1       <none>        443/TCP        2d22h
$ kubectl expose deployment nginx --port 80 --type LoadBalancer
service/nginx exposed
$ kubectl get service
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP      10.96.0.1       <none>        443/TCP        2d22h
nginx        LoadBalancer   10.107.27.81   <pending>     80:30635/TCP   4s

確認

$ curl 10.107.27.81
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

お掃除

ロードバランサ service 削除。

$ kubectl get service
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP      10.96.0.1       <none>        443/TCP        2d22h
nginx        LoadBalancer   10.103.26.220   <pending>     80:31747/TCP   13s
$ kubectl delete service nginx
service "nginx" deleted
$ kubectl get service
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   2d23h

nginx pod 削除。deployment を削除して pod が消えるのを待つ。

$ kubectl get deployment
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   1/1     1            1           3m41s
$ kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
nginx-86f6c4996d-s77wf   1/1     Running   0          3m47s
$ kubectl delete deployment nginx
deployment.extensions "nginx" deleted
$ kubectl get deployment
No resources found.
$ kubectl get pods
No resources found.