把Elasticsearch作为时间序列数据库使用

这篇文章算是对另一篇《Elasticsearch as a Time Series Data Store》的简单翻译吧，自己的理解吧。

首先_source被关闭了，这样原始的json文档不会被重复存储一遍。

其次_all也被关闭了。而且每个字段的store都是False，也就是不会单独被存储。

这些都关掉了，那么数据存哪里了？存在doc_values里。doc_values用于在做聚合运算的时候，根据一批文档id快速找到对应的列的值。doc_values在磁盘上一个按列压缩存储的文件，非常高效。

curl -XPOST http://172.16.18.116:9200/test -d '
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0,
"index.query.default_field": "timestamp",
"index.mapping.ignore_malformed": false,
"index.mapping.coerce": false,
"index.query.parse.allow_unmapped_fields": false
},
"mappings": {
"test": {
"_source": {"enabled": false},
"_all": {"enabled": false},
"properties": {
"timestamp": {
"type": "date",
"index": "no",
"store": false,
"dynamic": "strict",
"doc_values": true,
"fielddata": {
"format": "doc_values"
}
},
"appid": {
"type": "string",
"index": "no",
"store": false,
"dynamic": "strict",
"doc_values": true,
"fielddata": {
"format": "doc_values"
}
},
"result": {
"type": "string",
"index": "no",
"store": false,
"dynamic": "strict",
"doc_values": true,
"fielddata": {
"format": "doc_values"
}
},
"cmdid": {
"type": "string",
"index": "no",
"store": false,
"dynamic": "strict",
"doc_values": true,
"fielddata": {
"format": "doc_values"
}
},
"optime": {
"type": "integer",
"index": "no",
"store": false,
"dynamic": "strict",
"doc_values": true,
"fielddata": {
"format": "doc_values"
}
},
"total_count": {
"type": "integer",
"index": "no",
"store": false,
"dynamic": "strict",
"doc_values": true,
"fielddata": {
"format": "doc_values"
}
}
}
}
}
}'

增加一条数据：

curl -XPOST http://172.16.18.116:9200/test/test/1 -d '
{
"timestamp": 53534543,
"appid": 1,
"result": "test",
"cmdid": "test",
"optime": 53534543,
"total_count": 100
}
'

查询一下：

curl -XGET http://172.16.18.116:9200/test/test/_search {
"took": 1,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "test",
"_type": "test",
"_id": "1",
"_score": 1
}
]
}
}

能查到数据，但是看不到原始字段内容，因为没存储也没索引，但是doc_values=true，实际上是保存到了磁盘上的

下面做一下聚合操作：

curl -XPOST http://172.16.18.116:9200/test/test/_search {
"aggs": {
"timestamp": {
"terms": {
"field": "timestamp"
},
"aggs": {
"total_count": {"sum": {"field": "total_count"}}
}
}
}
}

结果：

{
"took": 2,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "test",
"_type": "test",
"_id": "1",
"_score": 1
}
]
},
"aggregations": {
"timestamp": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": 53534543,
"key_as_string": "1970-01-01T14:52:14.543Z",
"doc_count": 1,
"total_count": {
"value": 100
}
}
]
}
}
}

可以看到聚合操作可以获取到total_count值。