1、日志格式

{"10190":0,"10071":0,"10191":0,"10070":0,"48":"136587","type":"136587","10018":0}

我们如果收集这个日志只是做简单的配置。如下：

input {    file {        path => ["/home/elk/logstash-5.6.3/request"]        type => "chenxun"    }}output {    stdout {        codec => rubydebug    }    elasticsearch {        hosts => "192.168.2.181:9200"    }}

那么收集到的结果是：

{    "_index": "logstash-2017.11.22",    "_type": "chenxun",    "_id": "AV_iTR0AM1H1mf2je0nC",    "_version": 1,    "_score": 1,    "_source": {        "@version": "1",        "host": "Ubuntu-20170424",        "path": "/home/elk/logstash-5.6.3/request",        "@timestamp": "2017-11-22T05:57:05.383Z",        "message": "{"10190":0,"10071":0,"10191":0,"10070":0,"48":"136587","type":"136587","10018":0}",        "type": "chenxun"    }}

即会将json记录做为一个字符串放到”message”下,这不是我们想要的结果，是让logstash自动解析json记录，将各字段放入elasticsearch中。下面介绍如何配置.

1.直接设置codec => json

input {    file {        path => ["/home/elk/logstash-5.6.3/request"]        type => "chenxun"        codec => json    }   }

这个时候看看结果： 已经把json解析到各个字段中去了

{    "_index": "logstash-2017.11.22",    "_type": "136587",    "_id": "AV_iXHbGM1H1mf2jfF4d",    "_version": 1,    "_score": 1,    "_source": {        "48": "136587",        "10018": 0,        "10070": 0,        "10071": 0,        "10190": 0,        "10191": 0,        "path": "/home/elk/logstash-5.6.3/request",        "@timestamp": "2017-11-22T06:13:51.361Z",        "@version": "1",        "host": "Ubuntu-20170424",        "type": "136587"    }}

可以设置编码格式：（收集中文日志）

codec => json {            charset => "UTF-8"        }

2、使用filter json

配置如下：

input {    file {        path => ["/home/elk/logstash-5.6.3/request"]    }}filter {        json {            source => "message"            #target => "doc"            #remove_field => ["message"]        }        }output {    stdout {        codec => rubydebug    }    elasticsearch {        hosts => "192.168.2.181:9200"    }}

输入结果：

{    "_index": "logstash-2017.11.22",    "_type": "136587",    "_id": "AV_igupKM1H1mf2jfxm2",    "_version": 1,    "_score": 1,    "_source": {        "48": "136587",        "10018": 0,        "10070": 0,        "10071": 0,        "10190": 0,        "10191": 0,        "path": "/home/elk/logstash-5.6.3/request",        "@timestamp": "2017-11-22T06:55:51.335Z",        "@version": "1",        "host": "Ubuntu-20170424",        "message": "{"10190":0,"10071":0,"10191":0,"10070":0,"48":"136587","type":"136587","10018":0}",    "type": "136587"    }}

可以看到，原始记录被保存，同时字段也被解析保存。如果确认不需要保存原始记录内容，可以加设置：remove_field => [“message”]

其中特别需要注意解析json数据的内容，logstash会在向es插入数据时默认会在_source下增加type,host,path三个字段，如果json内容中本身也含有type,host,path字段，那么解析后将覆盖掉logstash默认的这三个字段，尤其是type字段，这个同时也是做为index/type用的，覆盖掉后，插入进es中的index/type就是json数据记录中的内容，将不再是logstash config中配置的type值。