kube-proxy 모니터링 환경 구성 실습(serviceMonitor, podMonitor)

가시다님의 Kubernetes Advanced Networking Study에 참여하게 되어, 스터디 때 다룬 주제를 정리하려고 합니다.
5주차는 LoadBalancer(MetalLB), IPVS + LoxiLB를 주제로 진행되었습니다.

이번 글에서는 kube-proxy 모니터링 환경 구성 실습에 대해 다루고 있습니다.

아래 포함된 테스트는 kind로 구성한 클러스터에서 진행되었습니다.

클러스터 생성

# EC2 생성
curl -O https://s3.ap-northeast-2.amazonaws.com/cloudformation.cloudneta.net/kans/kans-5w.yaml

aws cloudformation deploy --template-file kans-5w.yaml --stack-name mylab --parameter-overrides KeyName=<My SSH Keyname> SgIngressSshCidr=<My Home Public IP Address>/32 --region ap-northeast-2

# EC2 SSH 접속 후 kind 클러스터 생성
cat <<EOT> kind-svc-2w.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  "InPlacePodVerticalScaling": true  #실행 중인 파드의 리소스 요청 및 제한을 변경할 수 있게 합니다.
  "MultiCIDRServiceAllocator": true  #서비스에 대해 여러 CIDR 블록을 사용할 수 있게 합니다.
nodes:
- role: control-plane
  labels:
    mynode: control-plane
    topology.kubernetes.io/zone: ap-northeast-2a
  extraPortMappings:  #컨테이너 포트를 호스트 포트에 매핑하여 클러스터 외부에서 서비스에 접근할 수 있도록 합니다.
  - containerPort: 30000
    hostPort: 30000
  - containerPort: 30001
    hostPort: 30001
  - containerPort: 30002
    hostPort: 30002
  - containerPort: 30003
    hostPort: 30003
  - containerPort: 30004
    hostPort: 30004
  kubeadmConfigPatches:
  - |
    kind: ClusterConfiguration
    apiServer:
      extraArgs:  #API 서버에 추가 인수를 제공
        runtime-config: api/all=true  #모든 API 버전을 활성화
    controllerManager:
      extraArgs:
        bind-address: 0.0.0.0
    etcd:
      local:
        extraArgs:
          listen-metrics-urls: http://0.0.0.0:2381
    scheduler:
      extraArgs:
        bind-address: 0.0.0.0
  - |
    kind: KubeProxyConfiguration
    metricsBindAddress: 0.0.0.0
- role: worker
  labels:
    mynode: worker1
    topology.kubernetes.io/zone: ap-northeast-2a
- role: worker
  labels:
    mynode: worker2
    topology.kubernetes.io/zone: ap-northeast-2b
- role: worker
  labels:
    mynode: worker3
    topology.kubernetes.io/zone: ap-northeast-2c
networking:
  podSubnet: 10.10.0.0/16  #파드 IP를 위한 CIDR 범위를 정의합니다. 파드는 이 범위에서 IP를 할당받습니다.
  serviceSubnet: 10.200.1.0/24  #서비스 IP를 위한 CIDR 범위를 정의합니다. 서비스는 이 범위에서 IP를 할당받습니다.
EOT

# k8s 클러스터 설치
kind create cluster --config kind-svc-2w.yaml --name myk8s --image kindest/node:v1.31.0
docker ps

# 노드에 필요한 툴 설치
docker exec -it myk8s-control-plane sh -c 'apt update && apt install tree psmisc lsof wget bsdmainutils bridge-utils net-tools dnsutils ipset ipvsadm nfacct tcpdump ngrep iputils-ping arping git vim arp-scan -y'
for i in worker worker2 worker3; do echo ">> node myk8s-$i <<"; docker exec -it myk8s-$i sh -c 'apt update && apt install tree psmisc lsof wget bsdmainutils bridge-utils net-tools dnsutils ipset ipvsadm nfacct tcpdump ngrep iputils-ping arping -y'; echo; done

아래와 같이 클러스터를 생성합니다.

(⎈|kind-myk8s:N/A) root@MyServer:~# k get no
NAME                  STATUS   ROLES           AGE   VERSION
myk8s-control-plane   Ready    control-plane   24m   v1.31.0
myk8s-worker          Ready    <none>          24m   v1.31.0
myk8s-worker2         Ready    <none>          24m   v1.31.0
myk8s-worker3         Ready    <none>          24m   v1.31.0

 

kube-prometheus-stack 설치

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

# value 파일 생성
cat <<EOT > monitor-values.yaml
prometheus:
  service:
    type: NodePort
    nodePort: 30001

  prometheusSpec:
    podMonitorSelectorNilUsesHelmValues: false
    serviceMonitorSelectorNilUsesHelmValues: false
    nodeSelector:
      mynode: control-plane
    tolerations:
    - key: "node-role.kubernetes.io/control-plane"
      operator: "Equal"
      effect: "NoSchedule"


grafana:
  defaultDashboardsTimezone: Asia/Seoul
  adminPassword: kans1234

  service:
    type: NodePort
    nodePort: 30002
  nodeSelector:
    mynode: control-plane
  tolerations:
  - key: "node-role.kubernetes.io/control-plane"
    operator: "Equal"
    effect: "NoSchedule"

defaultRules:
  create: false
alertmanager:
  enabled: false
  
EOT
 
# kube-prometheus-stack 설치
kubectl create ns monitoring
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack --version 62.3.0 -f monitor-values.yaml --namespace monitoring

# 설치 후 확인
(⎈|kind-myk8s:N/A) root@MyServer:~# k get po -n monitoring
NAME                                                        READY   STATUS    RESTARTS   AGE
kube-prometheus-stack-grafana-789cdf5578-2jxp6              3/3     Running   0          22m
kube-prometheus-stack-kube-state-metrics-5689dc5579-d8lkz   1/1     Running   0          22m
kube-prometheus-stack-operator-8b86b65dc-4tsz4              1/1     Running   0          22m
kube-prometheus-stack-prometheus-node-exporter-2r5kh        1/1     Running   0          22m
kube-prometheus-stack-prometheus-node-exporter-jwz55        1/1     Running   0          22m
kube-prometheus-stack-prometheus-node-exporter-rk9jf        1/1     Running   0          22m
kube-prometheus-stack-prometheus-node-exporter-v85hj        1/1     Running   0          22m
prometheus-kube-prometheus-stack-prometheus-0               2/2     Running   0          21m

# prometheus, grafana url
# grafana 접속 계정 : admin / kans1234
echo -e "Prometheus URL = http://$(curl -s ipinfo.io/ip):30001"
echo -e "Grafana URL = http://$(curl -s ipinfo.io/ip):30002"

prometheus에 접속하여 kube-proxy 메트릭이 정상적으로 discover되고 있는지 확인할 수 있습니다.

차트 내 value.yaml 설정에 의해 service, serviceMonitor가 자동으로 생성되어 discover가 가능한 상태입니다.

(⎈|kind-myk8s:N/A) root@MyServer:~# k get servicemonitor -A
NAMESPACE    NAME                                             AGE
monitoring   kube-prometheus-stack-apiserver                  26m
monitoring   kube-prometheus-stack-coredns                    26m
monitoring   kube-prometheus-stack-grafana                    26m
monitoring   kube-prometheus-stack-kube-controller-manager    26m
monitoring   kube-prometheus-stack-kube-etcd                  26m
monitoring   kube-prometheus-stack-kube-proxy                 26m
monitoring   kube-prometheus-stack-kube-scheduler             26m
monitoring   kube-prometheus-stack-kube-state-metrics         26m
monitoring   kube-prometheus-stack-kubelet                    26m
monitoring   kube-prometheus-stack-operator                   26m
monitoring   kube-prometheus-stack-prometheus                 26m
monitoring   kube-prometheus-stack-prometheus-node-exporter   26m

(⎈|kind-myk8s:N/A) root@MyServer:~# k get svc -n kube-system
NAME                                            TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                        AGE
kube-dns                                        ClusterIP   10.200.1.10   <none>        53/UDP,53/TCP,9153/TCP         32m
kube-prometheus-stack-coredns                   ClusterIP   None          <none>        9153/TCP                       28m
kube-prometheus-stack-kube-controller-manager   ClusterIP   None          <none>        10257/TCP                      28m
kube-prometheus-stack-kube-etcd                 ClusterIP   None          <none>        2381/TCP                       28m
kube-prometheus-stack-kube-proxy                ClusterIP   None          <none>        10249/TCP                      28m
kube-prometheus-stack-kube-scheduler            ClusterIP   None          <none>        10259/TCP                      28m
kube-prometheus-stack-kubelet                   ClusterIP   None          <none>        10250/TCP,10255/TCP,4194/TCP   27m

(⎈|kind-myk8s:N/A) root@MyServer:~# k get svc kube-prometheus-stack-kube-proxy -n kube-system -oyaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: kube-prometheus-stack
    meta.helm.sh/release-namespace: monitoring
  creationTimestamp: "2024-10-16T13:00:45Z"
  labels:
    app: kube-prometheus-stack-kube-proxy
    app.kubernetes.io/instance: kube-prometheus-stack
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: kube-prometheus-stack
    app.kubernetes.io/version: 62.3.0
    chart: kube-prometheus-stack-62.3.0
    heritage: Helm
    jobLabel: kube-proxy
    release: kube-prometheus-stack
  name: kube-prometheus-stack-kube-proxy
  namespace: kube-system
  resourceVersion: "1207"
  uid: 7d20a9c8-6147-4a68-a908-4f21aa8ce034
spec:
  clusterIP: None
  clusterIPs:
  - None
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: http-metrics
    port: 10249
    protocol: TCP
    targetPort: 10249
  selector:
    k8s-app: kube-proxy
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}
  
(⎈|kind-myk8s:N/A) root@MyServer:~# k get servicemonitor kube-prometheus-stack-kube-proxy -n monitoring -oyaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    meta.helm.sh/release-name: kube-prometheus-stack
    meta.helm.sh/release-namespace: monitoring
  creationTimestamp: "2024-10-16T13:00:45Z"
  generation: 1
  labels:
    app: kube-prometheus-stack-kube-proxy
    app.kubernetes.io/instance: kube-prometheus-stack
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: kube-prometheus-stack
    app.kubernetes.io/version: 62.3.0
    chart: kube-prometheus-stack-62.3.0
    heritage: Helm
    release: kube-prometheus-stack
  name: kube-prometheus-stack-kube-proxy
  namespace: monitoring
  resourceVersion: "1304"
  uid: 11cc6c4b-27f5-4cd8-b9b5-a20b2cfb383e
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    port: http-metrics
  jobLabel: jobLabel
  namespaceSelector:
    matchNames:
    - kube-system
  selector:
    matchLabels:
      app: kube-prometheus-stack-kube-proxy
      release: kube-prometheus-stack

이번에는 podMonitor로 설정을 변경해보겠습니다.

먼저 기존 monitor-values.yaml 파일을 아래와 같이 수정합니다.

prometheus:
  service:
    type: NodePort
    nodePort: 30001

  prometheusSpec:
    podMonitorSelectorNilUsesHelmValues: false
    serviceMonitorSelectorNilUsesHelmValues: false
    nodeSelector:
      mynode: control-plane
    tolerations:
    - key: "node-role.kubernetes.io/control-plane"
      operator: "Equal"
      effect: "NoSchedule"
      
kubeProxy:
  serviceMonitor:
    enabled: false

grafana:
  defaultDashboardsTimezone: Asia/Seoul
  adminPassword: kans1234

  service:
    type: NodePort
    nodePort: 30002
  nodeSelector:
    mynode: control-plane
  tolerations:
  - key: "node-role.kubernetes.io/control-plane"
    operator: "Equal"
    effect: "NoSchedule"

defaultRules:
  create: false
alertmanager:
  enabled: false

그리고 helm upgrade를 진행합니다.

helm upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack --version 62.3.0 -f monitor-values.yaml --namespace monitoring
# serviceMonitor 삭제 확인
(⎈|kind-myk8s:N/A) root@MyServer:~# k get servicemonitor -A | grep proxy

다음으론 podMonitor를 생성합니다.

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  labels:
    app.kubernetes.io/instance: kube-proxy
    app.kubernetes.io/name: kube-proxy
    k8s-app: kube-proxy
  name: kube-proxy
  namespace: monitoring
spec:
  namespaceSelector:
    matchNames:
    - kube-system
  podMetricsEndpoints:
  - honorLabels: true
    interval: 1m
    path: /metrics
    port: metrics
    scrapeTimeout: 10s
  selector:
    matchLabels:
      k8s-app: kube-proxy

(⎈|kind-myk8s:N/A) root@MyServer:~# kubectl get podmonitor -n monitoring
NAME         AGE
kube-proxy   12m

그리고 기존 kube-proxy daemonset에 metrics 포트 설정을 추가합니다.

kubectl edit ds kube-proxy -n kube-system
apiVersion: apps/v1
kind: DaemonSet
metadata:
  annotations:
    deprecated.daemonset.template.generation: "2"
  creationTimestamp: "2024-10-16T12:55:52Z"
  generation: 2
  labels:
    k8s-app: kube-proxy
  name: kube-proxy
  namespace: kube-system
  resourceVersion: "6080"
  uid: c59812e1-74d5-4a67-9acb-6ad691e0fe67
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: kube-proxy
  template:
    metadata:
      creationTimestamp: null
      labels:
        k8s-app: kube-proxy
    spec:
      containers:
      - command:
        - /usr/local/bin/kube-proxy
        - --config=/var/lib/kube-proxy/config.conf
        - --hostname-override=$(NODE_NAME)
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        image: registry.k8s.io/kube-proxy:v1.31.0
        imagePullPolicy: IfNotPresent
        name: kube-proxy
        ## 해당 설정 추가 ##
        ports:
        - containerPort: 10249
          name: metrics
          protocol: TCP
...

이제 다시 prometheus에서 확인해보면, podMonitor를 통해 discover 되는 것을 확인할 수 있습니다.

podMonitor는 service가 없는 파드의 metric을 수집할 때 활용할 수 있습니다.

 

grafana에 접속하면 metric을 확인할 수 있습니다.

(참고로 metric의 job label의 value가 변경되어 변수 및 패널 수정이 필요합니다. job="monitoring/kube-proxy")

 

---

iptables-exporter 설치

kube-proxy에서 iptables를 사용하기 때문에, iptables 관련 메트릭을 수집할 수 있도록 설정해보겠습니다.

kind 환경에서 수행 시 권한 획득 등에 문제가 있기 때문에 EC2로 별도로 구성한 클러스터에서 테스트를 진행합니다.

(⎈|HomeLab:N/A) root@k8s-m:~# k get no -owide
NAME     STATUS   ROLES           AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME
k8s-m    Ready    control-plane   109m   v1.30.5   192.168.10.10    <none>        Ubuntu 22.04.5 LTS   6.8.0-1015-aws   containerd://1.7.22
k8s-w0   Ready    <none>          108m   v1.30.5   192.168.20.100   <none>        Ubuntu 22.04.5 LTS   6.8.0-1015-aws   containerd://1.7.22
k8s-w1   Ready    <none>          108m   v1.30.5   192.168.10.101   <none>        Ubuntu 22.04.5 LTS   6.8.0-1015-aws   containerd://1.7.22
k8s-w2   Ready    <none>          108m   v1.30.5   192.168.10.102   <none>        Ubuntu 22.04.5 LTS   6.8.0-1015-aws   containerd://1.7.22

참고 : https://github.com/kbknapp/iptables_exporter

GitHub - kbknapp/iptables_exporter: A Prometheus exporter for iptables in Rust

git clone https://github.com/kbknapp/iptables_exporter
cd iptables_exporter
apt install cargo -y
cargo build --release
cp target/release/iptables_exporter /usr/local/bin/

설치 후 iptables_exporter를 노드에서 실행합니다.

iptables_exporter -t iptables -t iptables-legacy -t ip6tables
2024-10-16T12:10:27.848285Z  INFO iptables_exporter: Registering metrics...
2024-10-16T12:10:27.848538Z  INFO iptables_exporter: Spawning server...
2024-10-16T12:10:27.848641Z  INFO iptables_exporter: Collecting iptables metrics...

이후 아래와 같이 노드에서 curl을 실행하면 metric이 발생하는 것을 확인할 수 있습니다.

(⎈|HomeLab:N/A) root@k8s-m:~# curl localhost:9455/metrics
# HELP ip6tables_chain_bytes_total Total bytes flowing through a given chain
# TYPE ip6tables_chain_bytes_total counter
ip6tables_chain_bytes_total{chain="FORWARD",policy="ACCEPT",table="filter"} 0
ip6tables_chain_bytes_total{chain="FORWARD",policy="ACCEPT",table="mangle"} 0
ip6tables_chain_bytes_total{chain="INPUT",policy="ACCEPT",table="filter"} 0
ip6tables_chain_bytes_total{chain="INPUT",policy="ACCEPT",table="mangle"} 0
ip6tables_chain_bytes_total{chain="INPUT",policy="ACCEPT",table="nat"} 0
...

9455 포트에 대해 metric을 scrap하기 위해 아래와 같이 prometheus 설정을 변경합니다.

prometheus:
  service:
    type: NodePort
    nodePort: 30001

  prometheusSpec:
    podMonitorSelectorNilUsesHelmValues: false
    serviceMonitorSelectorNilUsesHelmValues: false
    nodeSelector:
      kubernetes.io/hostname: k8s-w0
    tolerations:
    - key: "node-role.kubernetes.io/control-plane"
      operator: "Equal"
      effect: "NoSchedule"
    - operator: "Exists"
      effect: "NoSchedule"
    additionalScrapeConfigs:
      - job_name: 'iptables'
        static_configs:
        - targets:
          - '192.168.10.10:9455'
          - '192.168.20.100:9455'
        relabel_configs:
        - source_labels: [ '__address__' ]
          regex: '(.*):\d+'
          target_label: instance

nodeExporter:
  enabled: true

grafana:
  defaultDashboardsTimezone: Asia/Seoul
  adminPassword: kans1234

  service:
    type: NodePort
    nodePort: 30002
  nodeSelector:
    kubernetes.io/hostname: k8s-w0
  tolerations:
  - key: "node-role.kubernetes.io/control-plane"
    operator: "Equal"
    effect: "NoSchedule"

defaultRules:
  create: false
alertmanager:
  enabled: false

metric이 정상적으로 수집되는 것을 확인할 수 있습니다.

현재는 iptables_exporter를 노드에서 수동으로 실행했기 때문에, target에 iptables_exporter를 실행한 노드를 직접적으로 설정했습니다.

실제 사용하려면 daemonset으로 배포가 필요합니다.