AWS. EKS 애플리케이션 로그 수집하기 (3)
created Sep 11, 2022 | updated Oct 15, 2022
AWS EKS의 애플리케이션 로그를 AWS OpenSearch로 수집하고 관리하기 위한 내용을 정리한다. 로그 전송은 Fluent Bit을 사용했다.
내용은 간단하지만 한 개의 포스팅에 포함하려니 내용이 길어지고 가독성을 해쳐서 몇 개의 파트로 나눴다.
본 포스팅은 Fluent Bit DaemonSet을 설정하는 내용을 다룬다.
파트 목차
- 로그 수집 구성 개요
- AWS IRSA, K8S RBAC 설정
- DaemonSet 설정 : 현재 포스트
- 로그 로테이트 설정
- 로그 멀티라인 파싱 설정
Fluent Bit DaemonSet 설정
OpenSearch 접근 허용 설정
- DaemonSet에서 OpenSearch로 로그를 전송할 수 있도록 접근 허용 설정을 한다.
- ${OPEN_SEARCH_MASTER_USER}, ${OPEN_SEARCH_MASTER_PASSWORD}는 OpenSearch의 master user 계정이다. 이 계정은 fine-grained access control을 활성화하고 master user를 생성하면 만들 수 있다.
# 1) VPC 내부로 접속한다. 예를 들면, VPC 내 배스천 서버로 접속한다.
# 2) 변수 설정
export CLUSTER_NAME=logging-test-eks
export LOGGING_ROLE=$(kubectl get sa fluent-bit -n logging -o json |jq '.metadata.annotations."eks.amazonaws.com/role-arn"' -r)
export OPEN_SEARCH_ENDPOINT=https://vpc-logging-test-abcdefghijklmnopqrstuvwxyz.ap-southeast-1.es.amazonaws.com
export OPEN_SEARCH_MASTER_USER=XXXXXX
export OPEN_SEARCH_MASTER_PASSWORD=XXXXXX
# 3) OpenSearch Internal Database 업데이트
curl -sS -u "${OPEN_SEARCH_MASTER_USER}:${OPEN_SEARCH_MASTER_PASSWORD}" \
-X PATCH https://${OPEN_SEARCH_ENDPOINT}/_opendistro/_security/api/rolesmapping/all_access?pretty \
-H 'Content-Type: application/json' \
-d '[{"op": "add", "path": "/backend_roles", "value": [ "'${LOGGING_ROLE}'" ]}]'
K8S ConfigMap 생성
- ${OPENSEARCH_ENDPOINT}, ${AWS_REGION}, ${INDEX_NAME} 값은 데몬셋 컨테이너의 환경 변수에서 설정한다.
apiVersion: v1
kind: ConfigMap
metadata:
labels:
k8s-app: fluent-bit
name: fluent-bit-config
namespace: logging
data:
fluent-bit.conf: |
[SERVICE]
Flush 1
Log_Level info
Daemon off
Parsers_File parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
@INCLUDE input-kubernetes.conf
@INCLUDE filter-kubernetes.conf
@INCLUDE output-opensearch.conf
input-kubernetes.conf: |
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
Parser docker
DB /var/log/flb_kube.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
filter-kubernetes.conf: |
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Kube_Tag_Prefix kube.var.log.containers.
Merge_Log On
Merge_Log_Key log_processed
K8S-Logging.Parser On
K8S-Logging.Exclude On
output-opensearch.conf: |
[OUTPUT]
Name es
Match *
Host ${OPENSEARCH_ENDPOINT} Port 443
TLS On
AWS_Auth On
AWS_Region ${AWS_REGION} Index ${INDEX_NAME} Replace_Dots On
parsers.conf: |
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
K8S DaemonSet 생성
- 2편에서 생성한 Service Account 이름을 spec.template.spec.ServiceAccountName에 적어준다.
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
k8s-app: fluent-bit-logging
kubernetes.io/cluster-service: "true"
version: v1
name: fluent-bit
namespace: logging
spec:
selector:
matchLabels:
k8s-app: fluent-bit-logging
template:
metadata:
annotations:
prometheus.io/path: /api/v1/metrics/prometheus
prometheus.io/port: "2020"
prometheus.io/scrape: "true"
labels:
k8s-app: fluent-bit-logging
kubernetes.io/cluster-service: "true"
version: v1
spec:
containers:
- env: - name: OPENSEARCH_ENDPOINT value: vpc-logging-test-abcdefghijklmnopqrstuvwxyz.ap-southeast-1.es.amazonaws.com - name: AWS_REGION value: ap-southeast-1 - name: INDEX_NAME value: eks-log name: fluent-bit
image: amazon/aws-for-fluent-bit:2.28.0 imagePullPolicy: Always
ports:
- containerPort: 2020
volumeMounts:
- mountPath: /var/log
name: varlog
- mountPath: /var/lib/docker/containers
name: varlibdockercontainers
readOnly: true
- mountPath: /fluent-bit/etc/
name: fluent-bit-config
serviceAccountName: fluent-bit terminationGracePeriodSeconds: 10
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
operator: Exists
- effect: NoExecute
operator: Exists
- effect: NoSchedule
operator: Exists
volumes:
- hostPath:
path: /var/log
name: varlog
- hostPath:
path: /var/lib/docker/containers
name: varlibdockercontainers
- configMap:
name: fluent-bit-config
name: fluent-bit-config