kubernetes(1.8.3)系列之搭建EFK(Fluentd+Elasticsearch+Kibana)

Posted on 2018-02-07(星期三) 15:54 in Data

docker会将stdout和stderr输出到节点的一个目录下,一般是/var/lib/docker/contianers(也可能是/opt/docker/containers/,根据安装方式不同而不同), kubernetes 默认会将容器的stdout和stderr录入node(minion)的/var/log/containers目录下(最后还是链接到docker的日志路径下),而kubernetes 组件的日志默认放置在/var/log目录下。

EFK搭建使用的日志架构:

EFK安装

EFK介绍

  • Fluentd 2.0.1:负责收集日志(或者用Logstash,相对比较耗内存)
  • Elasticsearch 5.5.1:存储日志并提供搜索
  • Kibana 5.5.1:负责日志查询和展示

官方地址: https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch

安装EFK插件

通过在每台node上部署一个以DaemonSet方式运行的fluentd来收集每台node上的日志。Fluentd将docker日志目录/var/lib/docker/containers 和 /var/log 目录挂载到Pod中,然后Pod会在node节点的 /var/log/pods 目录中创建新的目录,可以区别不同的容器日志输出,该目录下有一个日志文件链接到/var/lib/docker/contianers目录下的容器日志输出。

docker的日志目录可能因为安装方式不同而不同,也可能在 /opt/docker/containers/ 下。

从官方地址下载所需要的配置文件,默认Fluentd中是监控/var/lib/docker/containers目录的,如果不一样,修改fluentd-es-ds.yaml文件中的挂载路径即可,否则最后不会接收到日志,kibana无法创建初始化的索引模式,这个最后说明。

-rw-r--r-- 1 root root   382 11月  8 10:26 es-service.yaml
-rw-r--r-- 1 root root  2841 11月  8 10:26 es-statefulset.yaml
-rw-r--r-- 1 root root 13462 11月  8 10:26 fluentd-es-configmap.yaml
-rw-r--r-- 1 root root  2822 2月   5 18:18 fluentd-es-ds.yaml
-rw-r--r-- 1 root root  1111 11月  8 10:26 kibana-deployment.yaml
-rw-r--r-- 1 root root   354 11月  8 10:26 kibana-service.yaml

给 Node 设置标签

定义 DaemonSet fluentd-es-v2.0.1 时设置了 nodeSelector beta.kubernetes.io/fluentd-ds-ready=true ,所以需要在期望运行 fluentd 的 Node 上设置该标签;

[[email protected]_master184 efk]# kubectl get nodes
NAME        STATUS    AGE       VERSION
192.168.1.185   Ready     1d        v1.6.0

[[email protected]_master184 efk]# kubectl label nodes 192.168.1.185 beta.kubernetes.io/fluentd-ds-ready=true
node "192.168.1.185" labeled

给其他几台node打上同样的标签。

执行定义文件

在下载好的配置路径下, 此处是 /etc/kubernetes/efk, 执行

[[email protected]_master184 efk]# kubectl create -f .
service "elasticsearch-logging" created
serviceaccount "elasticsearch-logging" created
clusterrole "elasticsearch-logging" created
clusterrolebinding "elasticsearch-logging" created
statefulset "elasticsearch-logging" created
configmap "fluentd-es-config-v0.1.0" created
serviceaccount "fluentd-es" created
clusterrole "fluentd-es" created
clusterrolebinding "fluentd-es" created
daemonset "fluentd-es-v2.0.1" created
deployment "kibana-logging" created
service "kibana-logging" created

检查执行结果

[[email protected]_master184 efk]# kubectl get deployment -n kube-system|grep kibana
kibana-logging             1         1         1            1           1d

[[email protected]_master184 efk]# kubectl get pods -n kube-system|grep -E 'elasticsearch|fluentd|kibana'
elasticsearch-logging-0                    1/1       Running   4          1d
elasticsearch-logging-1                    1/1       Running   0          1d
fluentd-es-v2.0.1-8d67k                    1/1       Running   0          1d
fluentd-es-v2.0.1-hzjfw                    1/1       Running   0          1d
fluentd-es-v2.0.1-tkbbn                    1/1       Running   0          1d
fluentd-es-v2.0.1-vmhhf                    1/1       Running   0          1d
fluentd-es-v2.0.1-vzkwz                    1/1       Running   0          1d
kibana-logging-5d7cbc6b5d-fbd4r            1/1       Running   0          1d

[[email protected]_master184 efk]# kubectl get service  -n kube-system|grep -E 'elasticsearch|kibana'  
elasticsearch-logging   ClusterIP   10.254.45.109    <none>        9200/TCP            1d
kibana-logging          ClusterIP   10.254.103.74    <none>        5601/TCP            1d

kibana Pod 第一次启动时会用较长时间(10-20分钟)来优化和 Cache 状态页面,可以 tailf 该 Pod 的日志观察进度:

[root@kube_master184 efk]# kubectl logs kibana-logging-5d7cbc6b5d-fbd4r -n kube-system -f
{"type":"log","@timestamp":"2018-02-05T10:14:14Z","tags":["info","optimize"],"pid":1,"message":"Optimizing and caching bundles for graph, ml, kibana, stateSessionStorageRedirect, timelion and status_page. This may take a few minutes"}
{"type":"log","@timestamp":"2018-02-05T10:16:01Z","tags":["info","optimize"],"pid":1,"message":"Optimization of bundles for graph, ml, kibana, stateSessionStorageRedirect, timelion and status_page complete in 106.41 seconds"}
{"type":"log","@timestamp":"2018-02-05T10:16:01Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-02-05T10:16:01Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-02-05T10:16:01Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-02-05T10:16:01Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-02-05T10:16:01Z","tags":["reporting","warning"],"pid":1,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2018-02-05T10:16:01Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-02-05T10:16:07Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"yellow","message":"Status changed from yellow to yellow - No existing Kibana index found","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2018-02-05T10:16:07Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"yellow","message":"Status changed from yellow to yellow - No existing Kibana index found","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2018-02-05T10:16:07Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"yellow","message":"Status changed from yellow to yellow - No existing Kibana index found","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2018-02-05T10:16:07Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"yellow","message":"Status changed from yellow to yellow - No existing Kibana index found","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2018-02-05T10:16:08Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"green","message":"Status changed from yellow to green - Kibana index ready","prevState":"yellow","prevMsg":"No existing Kibana index found"}
{"type":"log","@timestamp":"2018-02-05T10:16:08Z","tags":["license","info","xpack"],"pid":1,"message":"Imported license information from Elasticsearch for [data] cluster: mode: trial | status: active | expiry date: 2018-03-07T10:15:05+00:00"}
{"type":"log","@timestamp":"2018-02-05T10:16:08Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"No existing Kibana index found"}
{"type":"log","@timestamp":"2018-02-05T10:16:08Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"No existing Kibana index found"}
{"type":"log","@timestamp":"2018-02-05T10:16:08Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"No existing Kibana index found"}
{"type":"log","@timestamp":"2018-02-05T10:16:12Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-02-05T10:16:12Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-02-05T10:16:12Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"yellow","message":"Status changed from green to yellow - Waiting for Elasticsearch","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2018-02-05T10:16:12Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2018-02-05T10:16:12Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-02-05T10:16:12Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-02-05T10:16:12Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-02-05T10:16:12Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-02-05T10:16:12Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-02-05T10:16:12Z","tags":["status","plugin:[email protected]","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-02-05T10:16:12Z","tags":["listening","info"],"pid":1,"message":"Server running at http://0:5601"}
{"type":"log","@timestamp":"2018-02-05T10:16:12Z","tags":["status","ui settings","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}

访问 kibana

  • 通过 kube-apiserver 访问:

    获取 Kibana 服务 URL。

    [[email protected]_master184 efk]# kubectl cluster-info
    Kubernetes master is running at https://127.0.0.1:6443
    Elasticsearch is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/elasticsearch-logging/proxy
    Heapster is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/heapster/proxy
    Kibana is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kibana-logging/proxy
    KubeDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kube-dns/proxy
    kubernetes-dashboard is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy
    monitoring-grafana is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy
    
    To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
    

    浏览器访问 URL: https://192.168.1.184:6443/api/v1/proxy/namespaces/kube-system/services/kibana-logging

  • 通过 kubectl proxy 访问:

    创建代理

     $ kubectl proxy --address='192.168.1.184' --port=9999 --accept-hosts='^*$'
     Starting to serve on 192.168.1.184:9999
    

    浏览器访问 URL:http://192.168.1.184:9999/api/v1/proxy/namespaces/kube-system/services/kibana-logging

初始化 kibana

在 Management -> Index Patterns 页面创建一个 index(相当于 mysql 中的一个 database),Index name or pattern 使用默认的 logstash-*,Time Filter field name 下拉选中@timestamp, 点击 Create, 过一会儿就可以在 Discover页面查询日志了;

最后, 日志收集部分,除了Fluentd和Logstash,还可以使用Filebeat来收集日志,使用方法可以参考使用Filebeat收集Kubernetes中的应用日志

遇到问题

1. unable to fetch mapping. do you have indices matching the pattern kibana.

如果你在这里发现最下方并没有变成Create按钮且出现如下错误,说明fluent并没有向elasticsearch中写入数据,所以,分别检查fluent、elasticsearch的配置和日志,一定是两者之间通信出现问题了。

本次错误是因为docker写的日志并不在fluentd-es-ds.yaml配置文件中默认配置的/var/lib/docker/containers路径下,而在/opt/docker/containers下,所以并没有找到对应的日志文件(在不经意间查看/var/log/pods/下的文件时,发现链接的是/opt/docker/containers/路径下的文件,解决的办法就简单了,删掉原来的deamon-set, 修改挂载路径重新启动fluent即可了)

[[email protected] ea29c93b-0a45-11e8-af3c-d4bed9a7f66c]# ll
total 0
lrwxrwxrwx 1 root root 161 Feb  5 15:26 fluentd-es_0.log -> /opt/docker/containers/866f239a95f8530de148e5f2a178c55e0bdc4e5a918ed762e134ab94b134981d/866f239a95f8530de148e5f2a178c55e0bdc4e5a918ed762e134ab94b134981d-json.log

还有一种可能是因为docker的log-driver的设置,需要是json-file格式的,docker默认就是这种格式,如果以前修改过,需要改成json-file, 具体修改方法参考 docker logging

2. 修改elasticsearch日志文件存储位置

下载的es-statefulset.yaml配置文件中,默认是把elasticsearch的日志存储放在了pod的empty volumns里了,因此官方的部署参考仅可用于试验,实际部署到生产环境时需要使用Persistent Volume。如存储到node具体路径或ceph等。

参考文档

Docker Configure logging drivers

Fluentd Plugins

Kubernetes Logging Architecture

Elasticsearch: 权威指南

安装EFK插件

Kibana 5.2 中文文档

fluentd配置文件语法

使用fluentd实现实时收集日志文件