filebeat 获取nginx日志 发送给ElasticSearch

目的：通过filebeat获取nginx日志，发送给ElasticSearch，filebeat可以解析json格式日志，所以设置nginx日志为json格式。

1、配置nginx配置文件

    log_format jsonTest '{"@timestamp":"$time_iso8601",'                  '"host":"$server_addr",'                  '"service":"nginxTest",'                  '"trace":"$upstream_http_ctx_transaction_id",'                  '"log":"log",'                  '"clientip":"$remote_addr",'                  '"remote_user":"$remote_user",'                  '"request":"$request",'                  '"http_user_agent":"$http_user_agent",'                  '"size":$body_bytes_sent,'                  '"responsetime":$request_time,'                  '"upstreamtime":"$upstream_response_time",'                  '"upstreamhost":"$upstream_addr",'                  '"http_host":"$host",'                  '"url":"$uri",'                  '"domain":"$host",'                  '"xff":"$http_x_forwarded_for",'                  '"referer":"$http_referer",'                  '"status":"$status"}';     access_log /var/log/nginx/access.log jsonTest;

定义jsonTest的json格式，其中trace是页面response headers的值 ctx-transaction-id，通过upstream_http_ctx_transaction_id可以获取头文件属性。

将日志输出到/var/log/nginx/access.log

2、配置filebeat配置文件

filebeat.prospectors:- type: log  paths:   - '/root/front/logs/*.log'  json.message_key: log  json.keys_under_root: trueoutput.elasticsearch:  hosts: ["*.*.*.*:9200"]

将日志传输给ElasticSearch