Pause Container 이해하기

가시다님의 Kubernetes Advanced Networking Study에 참여하게 되어, 스터디 때 다룬 주제를 정리하려고 합니다.
2주차는 K8S Flannel CNI & PAUSE를 주제로 진행되었으며,이번 글에서는 Pause Container를 다룹니다.

파드 생성 단계

파드가 생성될 때 kubelet은 namespace와 cgroup을 설정하고, pause 컨테이너를 실행합니다.

그리고 CNI는 namespace를 pause 컨테이너에 binding 한 이후에, application 컨테이너를 실행합니다.

[도서] Core Kubernetes - Figure 4.1 The processes involved in container startup

 

kubelet 로그를 보다보면, 아래와 같은 로그를 종종 발견할 수 있습니다.

Sep 02 20:27:26 ip-10-xx-xx-xx.ap-northeast-2.compute.internal kubelet[3116]: I0902 20:27:26.520190    3116 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test/nginx-deployment-1-764d858646-k7k6r"
Sep 02 20:27:27 ip-10-xx-xx-xx.ap-northeast-2.compute.internal kubelet[3116]: I0902 20:27:27.402313    3116 kubelet.go:2384] "SyncLoop (PLEG): event for pod" pod="test/nginx-deployment-1-764d858646-k7k6r" event=&{ID:75da1db7-18e3-4456-932c-aac529482ee0 Type:ContainerStarted Data:fd3de1c3324e9a64a94d3126c996ab990931e620b37eb11990ab7b4016332b85}
Sep 02 20:27:27 ip-10-xx-xx-xx.ap-northeast-2.compute.internal kubelet[3116]: E0902 20:27:27.455630    3116 remote_runtime.go:176] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"4fd47d54fa28a752862e1894b924cc2619b83524125470c247679e93fc235060\": plugin type=\"aws-cni\" name=\"aws-cni\" failed (add): add cmd: failed to assign an IP address to container"
Sep 02 20:27:27 ip-10-xx-xx-xx.ap-northeast-2.compute.internal kubelet[3116]: E0902 20:27:27.455701    3116 kuberuntime_sandbox.go:72] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"4fd47d54fa28a752862e1894b924cc2619b83524125470c247679e93fc235060\": plugin type=\"aws-cni\" name=\"aws-cni\" failed (add): add cmd: failed to assign an IP address to container" pod="kube-addon/ebs-csi-node-q7s8c"
Sep 02 20:27:27 ip-10-xx-xx-xx.ap-northeast-2.compute.internal kubelet[3116]: E0902 20:27:27.455726    3116 kuberuntime_manager.go:1127] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"4fd47d54fa28a752862e1894b924cc2619b83524125470c247679e93fc235060\": plugin type=\"aws-cni\" name=\"aws-cni\" failed (add): add cmd: failed to assign an IP address to container" pod="kube-addon/ebs-csi-node-q7s8c"
Sep 02 20:27:27 ip-10-xx-xx-xx.ap-northeast-2.compute.internal kubelet[3116]: E0902 20:27:27.455781    3116 pod_workers.go:1294] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"ebs-csi-node-q7s8c_kube-addon(fe18ad40-750e-47db-aa8e-762f02462de2)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"ebs-csi-node-q7s8c_kube-addon(fe18ad40-750e-47db-aa8e-762f02462de2)\\\": rpc error: code = Unknown desc = failed to setup network for sandbox \\\"4fd47d54fa28a752862e1894b924cc2619b83524125470c247679e93fc235060\\\": plugin type=\\\"aws-cni\\\" name=\\\"aws-cni\\\" failed (add): add cmd: failed to assign an IP address to container\"" pod="kube-addon/ebs-csi-node-q7s8c" podUID=fe18ad40-750e-47db-aa8e-762f02462de2

sandbox 라는 키워드가 공통적으로 많이 등장하는데, 이는 파드가 실행되는 격리된 공간을 의미합니다.

kubelet의 주요 요소 중 하나인 SyncPod가 새로운 파드를 생성하거나 삭제할 때, sandbox에 대해 생성/삭제 요청을 합니다.

// 참고 : https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kuberuntime/kuberuntime_manager.go#L1040

// SyncPod syncs the running pod into the desired pod by executing following steps:
//
//  1. Compute sandbox and container changes.
//  2. Kill pod sandbox if necessary.
//  3. Kill any containers that should not be running.
//  4. Create sandbox if necessary.
//  5. Create ephemeral containers.
//  6. Create init containers.
//  7. Resize running containers (if InPlacePodVerticalScaling==true)
//  8. Create normal containers.

---

Pause Coantainer란?

파드 생성 단계에서 언급된 pause 컨테이너에 대해 알아보겠습니다.

파드는 여러 컨테이너로 구성됩니다. 그리고 모든 파드에는 pause 컨테이너가 포함되어 있습니다.

pause 컨테이너는 아래와 같은 특징이 있습니다.

• Network/IPC/UTS namespace 생성/유지/공유
  
  • 파드 내 컨테이너는 pause 컨테이너가 생성한 network namespace 공유
  • 파드 내 컨테이너가 재시작해도 IP가 유지되는 이유는 pause 컨테이너가 network namespace를 유지하기 때문
• 파드의 전체 라이프사이클 동안 실행

실제 생성된 파드의 프로세스를 확인해보겠습니다.

[root@ip-10-xx-xx-xx /]# pstree -aln
systemd --switched-root --system --deserialize 21
  ├─systemd-journal
  ├─lvmetad -f
  ├─systemd-udevd
  ├─auditd
...
  ├─containerd-shim -namespace k8s.io -id fd3de1c3324e9a64a94d3126c996ab990931e620b37eb11990ab7b4016332b85 -address /run/containerd/containerd.sock
  │   ├─11*[{containerd-shim}]
  │   ├─pause
  │   └─nginx
  │       ├─nginx
  │       ├─nginx
  │       ├─nginx
  │       └─nginx
...
[root@ip-10-xx-xx-xx /]# pstree -aclnpsS
systemd,1 --switched-root --system --deserialize 21
  ├─systemd-journal,1390
  ├─lvmetad,1407 -f
  ├─systemd-udevd,1898
...
  ├─containerd-shim,4254 -namespace k8s.io -id fd3de1c3324e9a64a94d3126c996ab990931e620b37eb11990ab7b4016332b85 -address /run/containerd/containerd.sock
  │   ├─{containerd-shim},4256
  │   ├─{containerd-shim},4257
  │   ├─{containerd-shim},4258
  │   ├─{containerd-shim},4259
  │   ├─{containerd-shim},4260
  │   ├─{containerd-shim},4261
  │   ├─{containerd-shim},4262
  │   ├─{containerd-shim},4263
  │   ├─{containerd-shim},4264
  │   ├─{containerd-shim},4265
  │   ├─pause,4276,ipc,mnt,net,pid,uts
  │   ├─nginx,4901,ipc,mnt,net,pid,uts
  │   │   ├─nginx,4936
  │   │   ├─nginx,4937
  │   │   ├─nginx,4938
  │   │   └─nginx,4939
...

## node
[root@ip-10-xx-xx-xx /]# lsns -p 1
        NS TYPE   NPROCS PID USER COMMAND
4026531835 cgroup    158   1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 21
4026531836 pid       129   1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 21
4026531837 user      158   1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 21
4026531838 uts       138   1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 21
4026531839 ipc       123   1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 21
4026531840 mnt       121   1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 21
4026531992 net       138   1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 21

## containerd-shim
[root@ip-10-xx-xx-xx /]# lsns -p 4254
        NS TYPE   NPROCS PID USER COMMAND
4026531835 cgroup    153   1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 21
4026531836 pid       124   1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 21
4026531837 user      153   1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 21
4026531838 uts       133   1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 21
4026531839 ipc       118   1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 21
4026531840 mnt       116   1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 21
4026531992 net       133   1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 21

## pause
[root@ip-10-xx-xx-xx /]# lsns -p 4276
        NS TYPE   NPROCS   PID USER  COMMAND
4026531835 cgroup    153     1 root  /usr/lib/systemd/systemd --switched-root --system --deserialize 21
4026531837 user      153     1 root  /usr/lib/systemd/systemd --switched-root --system --deserialize 21
4026532379 net         6  4276 65535 /pause
4026532455 mnt         1  4276 65535 /pause
4026532456 uts         6  4276 65535 /pause
4026532457 ipc         6  4276 65535 /pause
4026532458 pid         1  4276 65535 /pause

## nginx
[root@ip-10-xx-xx-xx /]# lsns -p 4901
        NS TYPE   NPROCS   PID USER  COMMAND
4026531835 cgroup    153     1 root  /usr/lib/systemd/systemd --switched-root --system --deserialize 21
4026531837 user      153     1 root  /usr/lib/systemd/systemd --switched-root --system --deserialize 21
4026532379 net         6  4276 65535 /pause
4026532456 uts         6  4276 65535 /pause
4026532457 ipc         6  4276 65535 /pause
4026532584 mnt         5  4901 root  nginx: master process nginx -g daemon off;
4026532585 pid         5  4901 root  nginx: master process nginx -g daemon off;

containerd-shim에 아래에 pause, nginx 프로세스가 존재하고, pause와 nginx 프로세스의 namespace를 비교해보면, mnt와 pid namespace를 제외하고 모두 같은 namespace를 공유하는 것을 확인할 수 있습니다.

또한, pause 프로세스는 노드와 cgroup, user namespace를 공유합니다.

 

Pause Coantainer 이미지

pause 컨테이너를 생성하기에 앞서, 노드에 이미지가 없다면 이미지 레지스트리에서 다운받습니다.

참고로, EKS에서 kubelet 옵션을 살펴보면, pause 컨테이너 이미지가 포함된 것을 볼 수 있습니다.

[root@ip-10-xx-xx-xx /]# ps -ef | grep kubelet
root        3116       1  1 20:27 ?        00:00:54 /usr/bin/kubelet --config /etc/kubernetes/kubelet/kubelet-config.json --kubeconfig /var/lib/kubelet/kubeconfig --container-runtime-endpoint unix:///run/containerd/containerd.sock --image-credential-provider-config /etc/eks/image-credential-provider/config.json --image-credential-provider-bin-dir /etc/eks/image-credential-provider --node-ip=10.xx.xx.xx --pod-infra-container-image=602401143452.dkr.ecr.ap-northeast-2.amazonaws.com/eks/pause:3.5 --v=2 --hostname-override=ip-10-xx-xx-xx.ap-northeast-2.compute.internal --cloud-provider=external --node-labels=eks.amazonaws.com/capacityType=ON_DEMAND,xxx --max-pods=58 --system-reserved=ephemeral-storage=1Gi,memory=100Mi,cpu=100m --kube-reserved=memory=100Mi,cpu=200m,ephemeral-storage=3Gi --eviction-hard=nodefs.inodesFree<10%,memory.available<256Mi,nodefs.available<10% --eviction-soft=memory.available<500Mi,nodefs.available<15%,nodefs.inodesFree<15% --eviction-soft-grace-period=memory.available=1m0s,nodefs.available=1m30s,nodefs.inodesFree=2m0s --eviction-max-pod-grace-period=60

[이미지 설정]

--pod-infra-container-image=602401143452.dkr.ecr.ap-northeast-2.amazonaws.com/eks/pause:3.5

[root@ip-10-xx-xx-xx /]# grep sandbox /etc/containerd/config.toml
sandbox_image = "602401143452.dkr.ecr.ap-northeast-2.amazonaws.com/eks/pause:3.5"

 

또한, Amazon EKS Optimized AMI(al2 기준)에는 해당 이미지를 다운로드 받는 단계가 포함되어 있습니다(참고 - 문서, pull-sandbox-image.sh).

When container image caching is enabled, the following images are cached:
 - 602401143452.dkr.ecr.<AWS_REGION>.amazonaws.com/eks/kube-proxy:<default and latest>-eksbuild.<BUILD_VERSION>
 - 602401143452.dkr.ecr.<AWS_REGION>.amazonaws.com/eks/kube-proxy:<default and latest>-minimal-eksbuild.<BUILD_VERSION>
 - 602401143452.dkr.ecr.<AWS_REGION>.amazonaws.com/eks/pause:3.5
 - 602401143452.dkr.ecr.<AWS_REGION>.amazonaws.com/amazon-k8s-cni-init:<default and latest>
 - 602401143452.dkr.ecr.<AWS_REGION>.amazonaws.com/amazon-k8s-cni:<default and latest>

pause 컨테이너 이미지는 public ecr이 아닌 private ecr에 업로드되어 있기 때문에, 혹시라도 노드에서 pause 컨테이너 이미지가 삭제된다면 권한 이슈로 이미지를 다운로드 받지 못해서 파드 생성에 실패할 수 있습니다.

Warning  FailedCreatePodSandBox  6s (x8 over 79s)   kubelet                  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "602401143452.dkr.ecr.ap-northeast-2.amazonaws.com/eks/pause:3.5": failed to pull image "602401143452.dkr.ecr.ap-northeast-2.amazonaws.com/eks/pause:3.5": failed to pull and unpack image "602401143452.dkr.ecr.ap-northeast-2.amazonaws.com/eks/pause:3.5": failed to resolve reference "602401143452.dkr.ecr.ap-northeast-2.amazonaws.com/eks/pause:3.5": pull access denied, repository does not exist or may require authorization: authorization failed: no basic auth credentials

 

이런 경우에는 이슈가 발생한 노드에서 ECR 토큰을 발급 받고 이미지를 다운받으면 해결됩니다(노드의 iam role에 AmazonEC2ContainerRegistryReadOnly 권한 필요).

ctr -n k8s.io images pull --user "AWS:$(aws ecr get-login-password --region ap-northeast-2)" 602401143452.dkr.ecr.ap-northeast-2.amazonaws.com/eks/pause:3.5

 

참고로, cri-tool의 버전이 1.30보다 낮다면, crictl rmi --prune 명령어 실행 시 pause 컨테이너 이미지(pinned 이미지)가 삭제될 수 있습니다(참고- 이슈, 릴리즈노트).

 

참고)

• https://www.ithands-on.com/2020/11/kubernetes-101-pause-containers.html
• https://kubernetes.io/blog/2016/12/container-runtime-interface-cri-in-kubernetes/