AWS EKS의 애플리케이션 로그를 AWS OpenSearch로 수집하고 관리하기 위한 내용을 정리한다. 로그 전송은 Fluent Bit을 사용했다.

내용은 간단하지만 한 개의 포스팅에 포함하려니 내용이 길어지고 가독성을 해쳐서 몇 개의 파트로 나눴다.

본 포스팅은 Fluent Bit DaemonSet을 설정하는 내용을 다룬다.

파트 목차

Fluent Bit DaemonSet 설정

OpenSearch 접근 허용 설정

DaemonSet에서 OpenSearch로 로그를 전송할 수 있도록 접근 허용 설정을 한다.
${OPEN_SEARCH_MASTER_USER}, ${OPEN_SEARCH_MASTER_PASSWORD}는 OpenSearch의 master user 계정이다. 이 계정은 fine-grained access control을 활성화하고 master user를 생성하면 만들 수 있다.

# 1) VPC 내부로 접속한다. 예를 들면, VPC 내 배스천 서버로 접속한다.

# 2) 변수 설정
export CLUSTER_NAME=logging-test-eks
export LOGGING_ROLE=$(kubectl get sa fluent-bit -n logging -o json |jq '.metadata.annotations."eks.amazonaws.com/role-arn"' -r)
export OPEN_SEARCH_ENDPOINT=https://vpc-logging-test-abcdefghijklmnopqrstuvwxyz.ap-southeast-1.es.amazonaws.com
export OPEN_SEARCH_MASTER_USER=XXXXXX
export OPEN_SEARCH_MASTER_PASSWORD=XXXXXX

# 3) OpenSearch Internal Database 업데이트
curl -sS -u "${OPEN_SEARCH_MASTER_USER}:${OPEN_SEARCH_MASTER_PASSWORD}" \
    -X PATCH https://${OPEN_SEARCH_ENDPOINT}/_opendistro/_security/api/rolesmapping/all_access?pretty \
    -H 'Content-Type: application/json' \
    -d '[{"op": "add", "path": "/backend_roles", "value": [ "'${LOGGING_ROLE}'" ]}]'

K8S ConfigMap 생성

${OPENSEARCH_ENDPOINT}, ${AWS_REGION}, ${INDEX_NAME} 값은 데몬셋 컨테이너의 환경 변수에서 설정한다.

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    k8s-app: fluent-bit
  name: fluent-bit-config
  namespace: logging
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-opensearch.conf
  
  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10
  
  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On
  
  output-opensearch.conf: |
    [OUTPUT]
        Name            es
        Match           *
        Host            ${OPENSEARCH_ENDPOINT}        Port            443
        TLS             On
        AWS_Auth        On
        AWS_Region      ${AWS_REGION}        Index           ${INDEX_NAME}        Replace_Dots    On
        
  parsers.conf: |
    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

K8S DaemonSet 생성

2편에서 생성한 Service Account 이름을 spec.template.spec.ServiceAccountName에 적어준다.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    k8s-app: fluent-bit-logging
    kubernetes.io/cluster-service: "true"
    version: v1
  name: fluent-bit
  namespace: logging
spec:
  selector:
    matchLabels:
      k8s-app: fluent-bit-logging
  template:
    metadata:
      annotations:
        prometheus.io/path: /api/v1/metrics/prometheus
        prometheus.io/port: "2020"
        prometheus.io/scrape: "true"
      labels:
        k8s-app: fluent-bit-logging
        kubernetes.io/cluster-service: "true"
        version: v1
    spec:
      containers:
      - env:        - name: OPENSEARCH_ENDPOINT          value: vpc-logging-test-abcdefghijklmnopqrstuvwxyz.ap-southeast-1.es.amazonaws.com        - name: AWS_REGION          value: ap-southeast-1        - name: INDEX_NAME          value: eks-log        name: fluent-bit
        image: amazon/aws-for-fluent-bit:2.28.0        imagePullPolicy: Always
        ports:
        - containerPort: 2020
        volumeMounts:
        - mountPath: /var/log
          name: varlog
        - mountPath: /var/lib/docker/containers
          name: varlibdockercontainers
          readOnly: true
        - mountPath: /fluent-bit/etc/
          name: fluent-bit-config
      serviceAccountName: fluent-bit      terminationGracePeriodSeconds: 10
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
        operator: Exists
      - effect: NoExecute
        operator: Exists
      - effect: NoSchedule
        operator: Exists
      volumes:
      - hostPath:
          path: /var/log
        name: varlog
      - hostPath:
          path: /var/lib/docker/containers
        name: varlibdockercontainers
      - configMap:
          name: fluent-bit-config
        name: fluent-bit-config

참고

https://www.eksworkshop.com/intermediate/230_logging/deploy.files/fluentbit.yaml