关于fielddata数据占用内存过大的解决方法
2016-01-13 15:59
525 查看
参考文章
Support in the Wild: My Biggest Elasticsearch Problem at Scale
http://blog.csdn.net/jiao_fuyou/article/details/50478198Understanding Fielddata
By default, fielddata is loaded on demand, which means that you will not see it until you are using it. Also, by being loaded per segment, it means that new segments that get created will slowly add to your overall memory usage until the field’s fielddata is evicted from memory. Eviction happens in only a few ways:1、Deleting the index or indices that contains it.
2、Closing the index or indices that contains it.
3、Segment fielddata is removed when segments are removed (e.g., background merging).
* This usually just means that the problem is moving rather than going away.
4、Restarting the node containing the fielddata.
5、Clearing the relevant fielddata cache.
6、Automatically evicting the fielddata to make room for other fielddata.
* This will not happen with default settings.
Use Doc Values
The solution to this fielddata problem is to avoid it altogether. Fortunately, you can avoid the use of fielddata bymanually mapping all of your fields to use doc values. Without repeating too much from the guide, doc values offload this burden by writing the fielddata to disk at index time, thereby allowing Elasticsearch to load the values outside of your Java heap as they are needed.You might be asking: How do I enable these doc values? It’s as simple as adding this to every field’s mapping, exceptanalyzed string fields:
"doc_values" : true
Unfortunately, you must do this before you index any data. This means that, for any existing index that is not already using doc values, you cannot flip the switch to enable them. You would have to reindex.
Performance Considerations for Elasticsearch Indexing
http://blog.csdn.net/jiao_fuyou/article/details/50480663Limiting Memory Usage
https://www.elastic.co/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html#fielddata-sizeEnabled Doc Values
https://www.elastic.co/guide/en/elasticsearch/guide/current/doc-values.html控制fielddata允许内存大小
控制fielddata允许内存大小,达到HEAP 40% 自动清理旧cachecurl -XPUT http://localhost:9200/_cluster/settings { "persistent" : { "indices.fielddata.cache.size" : "40%" } }
如果旧缓存不需要,如日志,只查询最新一天数据,那可以强制清除旧缓存:
POST /_cache/clear
开启字段doc_values
对fielddata战用很大内存的字段,设置:”doc_values” : true,ES2.0以后,该属性默认开启。PUT /music/_mapping/song { "properties" : { "tag": { "type": "string", "index" : "not_analyzed", "doc_values": true } } }
相关文章推荐
- 巧用mysql提示符prompt清晰管理数据库的方法
- 两大步骤教您开启MySQL 数据库远程登陆帐号的方法
- phpmyadmin 4+ 访问慢的解决方法
- linux系统下实现mysql热备份详细步骤(mysql主从复制)
- CentOS 5.5下安装MySQL 5.5全过程分享
- MySQL复制的概述、安装、故障、技巧、工具(火丁分享)
- MySQL中删除重复数据的简单方法
- elasticsearch批量数据导入和导出
- 使用ElasticSearch+LogStash+Kibana+Redis搭建日志管理服务
- ElasticSearch 使用心得
- ELK(ElasticSearch, Logstash, Kibana)搭建实时日志分析平台
- CENTOS安装ElasticSearch
- ElasticSearch的安装配置
- 《Elasticsearch: The Definitive Guide》读书笔记2
- ElasticSearch REST API
- 网站基于ElasticSearch搜索的优化笔记 PHP
- elasticsearch安装与调试
- elasticsearch建立geo search
- elasticsearch在NAS上的部署和使用
- 关于ElasticSearch的一些肤浅的思考