ELK之LogStash读取JSON日志分类型建立索引

摘要: 测试目的是，用ELK处理在业务中用户定义的json log日志,则试PHP脚本如下

测试目的是，用ELK处理在业务中用户定义的json log日志,则试PHP脚本如下:

<?php
for( $i=0;$i<100;$i++)
{
$reg = array(
'method' => 'login',
'user_id' => rand(1000,3000),
'user_name' => "name_".rand(1,3000 ),
'level' => 1,
'register_time' => time(),
);
$str = json_encode( $reg );
file_put_contents( "testlog" , $str."\n" , FILE_APPEND );
$reg = array(
'method' => 'register',
'user_id' => rand(1000,3000),
'user_name' => "name_".rand(1,3000 ),
'level' => rand(1,30),
'login_time' => time(),
);
$str = json_encode( $reg );
file_put_contents( "testlog" , $str."\n" , FILE_APPEND );
}

循环生成注册log和登录log保存到testlog文件中,结果如下:

{"method":"register","user_id":2933,"user_name":"name_91","level":27,"login_time":1470179550}
{"method":"login","user_id":1247,"user_name":"name_979","level":1,"register_time":1470179550}
{"method":"register","user_id":2896,"user_name":"name_1972","level":17,"login_time":1470179550}
{"method":"login","user_id":2411,"user_name":"name_2719","level":1,"register_time":1470179550}
{"method":"register","user_id":1588,"user_name":"name_1484","level":4,"login_time":1470179550}
{"method":"login","user_id":2507,"user_name":"name_1190","level":1,"register_time":1470179550}
{"method":"register","user_id":2382,"user_name":"name_234","level":21,"login_time":1470179550}
{"method":"login","user_id":1208,"user_name":"name_443","level":1,"register_time":1470179550}
{"method":"register","user_id":1331,"user_name":"name_1297","level":3,"login_time":1470179550}
{"method":"login","user_id":2809,"user_name":"name_743","level":1,"register_time":1470179550}

logstash目录下建立配置文件

vim config/json.conf

input {
file {
path => "/home/bona/logstash-2.3.4/testlog"
start_position => "beginning"
codec => "json"
}
}

output {
elasticsearch {
hosts => ["192.168.68.135:9200"]
index => "data_%{method}"
}
}

重点是index中，%{method} 来匹配log中的method字段.

以上log就会分别建立data_login data_register两个索引, 要注意的是索引名称必须全部小写

ES中已经成功以method建立了索引



elasticsearch-sql查询



参考资料:
http://udn.yyuap.com/doc/logstash-best-practice-cn/output/elasticsearch.html https://github.com/NLPchina/elasticsearch-sql