Kubernetes

EFK Stack 구성하기 (Amazon OpenSearch + FluentBit + Kibana)

백곰곰 2022. 5. 23. 22:10
728x90
반응형

Kubernetes 내 컨테이너의 로그 수집에는 다양한 방법이 있습니다. 구성 조합에 따라 ELK, EFK 등이 있고, Amazon EKS 구성 시에는 Container Insights를 통해 CloudWatch에 로그를 게시할 수 있습니다.

 

그 중 AWS for Fluent Bit 이미지를 사용하지 못하는 경우에 fluentbit과 Amazon OpenSearch Service를 활용하여 EFK를 구성해보겠습니다. 현재 EKS Anywhere에서는 AWS for Fleunt Bit 이미지 사용 시 metadata를 찾지 못해 정상적으로 컨테이너를 구동할 수 없습니다.

 

[목표]

[기본 환경]

  • Internet Outbound 통신이 가능한 Kubernetes Cluster

[배포 순서]

1. IAM User 생성

resource "aws_iam_user" "efk-test" {
  name = "efk-test"
  path = "/"
  force_destroy = true
}

2. Amazon OpenSearch Domain 및 IAM Role 생성 [Terraform]

Kibana는 OpenSearch 생성 시 자동으로 생성됩니다.

테스트 시 인터넷 접근을 위해 VPC를 지정하지 않고 Public으로 생성합니다.

### Amazon OpenSearch ###
resource "aws_elasticsearch_domain" "log-opensearch" {
  domain_name           = "log-opensearch"
  elasticsearch_version = "7.10"

  cluster_config {
    instance_type  = "r4.large.elasticsearch"
    instance_count = 1
  }

  ebs_options {
    ebs_enabled = true
    volume_size = 10
  }

  access_policies = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "es:*",
      "Principal": "*",
      "Effect": "Allow",
      "Resource": "arn:aws:es:${var.region}:${var.aws-account-id}:domain/log-opensearch/*",
      "Condition": {
        "IpAddress": {"aws:SourceIp": [
             "1.1.1.1/32",      ## 접근 필요한 IP로 수정
             "2.2.2.0/24"
            ]
         }
      }
    }
  ]
}
POLICY

  tags = {
    Name = "log-opensearch"
  }
}

### IAM Role ###
resource "aws_iam_role" "role-efk-test" {
  name = "role-efk-test"
  
  assume_role_policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::${var.aws-account-id}:user/efk-test"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
EOF
}

resource "aws_iam_role_policy_attachment" "role-efk-test-policya-1" {
  role      = aws_iam_role.role-efk-test.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonOpenSearchServiceFullAccess"
}

 

3. fluentbit 배포 (Kubernetes 1.22 기준)

$ kubectl create namespace logging
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-service-account.yaml
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role-1.22.yaml
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role-binding-1.22.yaml

[fluent-bit-ds.yaml]

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
  labels:
    k8s-app: fluent-bit-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    matchLabels:
      k8s-app: fluent-bit-logging
  template:
    metadata:
      labels:
        k8s-app: fluent-bit-logging
        version: v1
        kubernetes.io/cluster-service: "true"
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "2020"
        prometheus.io/path: /api/v1/metrics/prometheus
    spec:
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:1.5
        imagePullPolicy: Always
        ports:
          - containerPort: 2020
        env:
        - name: FLUENT_ELASTICSEARCH_HOST
          value: "search-log-opensearch-xxxxxx.ap-northeast-2.es.amazonaws.com"  ##OpenSearch Domain Endpoint
        - name: FLUENT_ELASTICSEARCH_PORT
          value: "443"
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              name: secret-aws-credentials
              key: AWS_ACCESS_KEY_ID
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: secret-aws-credentials
              key: AWS_SECRET_ACCESS_KEY
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
      terminationGracePeriodSeconds: 10
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config
      serviceAccountName: fluent-bit
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - operator: "Exists"
        effect: "NoExecute"
      - operator: "Exists"
        effect: "NoSchedule"

[fluent-bit-configmap.yaml]

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
  labels:
    k8s-app: fluent-bit
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-elasticsearch.conf

  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10

  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off

  output-elasticsearch.conf: |
    [OUTPUT]
        Name            es
        Match           *
        Host            ${FLUENT_ELASTICSEARCH_HOST}
        Port            ${FLUENT_ELASTICSEARCH_PORT}
        Index           kubernetes-log
        AWS_Auth        On
        AWS_Region      ap-northeast-2
        aws_role_arn    arn:aws:iam::{aws-account-id}:role/role-efk-test
        Logstash_Format On
        Replace_Dots    On
        Retry_Limit     False
        tls             On
        tls.verify      Off

  parsers.conf: |
    [PARSER]
        Name   apache
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache2
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache_error
        Format regex
        Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$

    [PARSER]
        Name   nginx
        Format regex
        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

    [PARSER]
        # http://rubular.com/r/tjUt3Awgg4
        Name cri
        Format regex
        Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z

    [PARSER]
        Name        syslog
        Format      regex
        Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key    time
        Time_Format %b %d %H:%M:%S

 

[AWS Credential Secret]

apiVersion: v1
kind: Secret
metadata:
  name: secret-aws-credentials
  namespace: logging
type: Opaque
data:
  AWS_ACCESS_KEY_ID: sdfsdfwefwef=  ### efk-test IAM User Credentials base64로 인코딩하여 입력
  AWS_SECRET_ACCESS_KEY: WHsssTmtHVTJYRGsssZnRlaUpYTFhiV3VgggQzVXlybnZzzzI0czJheg==

[참고 - base64 인코딩 방법]

echo -n 'AWS_ACCESS_KEY_ID값' | base64
echo -n 'AWS_SECRET_ACCESS_KEY값' | base64

 

4. fluentbit DaemonSet 배포 확인

kubectl get po -n logging

5. Kibana 접속 후 로그 확인 (Kibana URL 클릭)

[추후 확인할 것]

  • Fluentbit - Kinesis 연동 가능 여부 (AWS for Fluent Bit 이미지 제외)
  • Kibana Index 상세 파악(logstash)
    • 왜 logstash로 만들어지는지?
      -> fluentbit 설정에 Logstash_Format: On로 설정되어있기 때문에 index가 logstash- 로 시작하는 이름의 index로 생성됨

[참고자료]

https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch

 

Elasticsearch - Fluent Bit: Official Manual

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status"

docs.fluentbit.io

 

728x90
저작자표시

'Kubernetes' 카테고리의 다른 글

EFS 권장 옵션을 포함한 EKS 내 PV 생성  (0) 2022.06.24
EFK Stack 구성하기 2 (Kinesis Firehose 추가)  (2) 2022.05.25
Amazon EKS Cluster Autoscaler OOM killed 현상  (0) 2022.05.08
Kubernetes 스토리지 관련 객체  (0) 2021.08.21
Kubernetes 워크로드 객체 특징  (0) 2021.08.19

'Kubernetes'의 다른글

  • 현재글EFK Stack 구성하기 (Amazon OpenSearch + FluentBit + Kibana)

관련글

댓글

티스토리툴바