Elastic Stack5.2.2升级到6.0.0注意事项
2018-01-19 15:46
761 查看
最近把Elastic Stack从5.2.2版本升级到6.0.0版本,性能确实有所提高,文档记录了升级过程中需要注意的一些问题。
架构图
一、Filebeat
6.0版本filebeat prospectors中的document_type被禁用,原来的topic: '%{[type]}'获取文档类型的方式不可用,但是引入了fields,后面映射模板也会用到。1、document_type: yewuname 原来的Index名字为filebeat-yewuname-2018.01.19这样的按照天分割的,升级后document_type不支持了,需要修改为字段加type的形式如下fields: document_type: yewuname
2、输出kafka也需要修改topic: '%{[fields.document_type]}'
3、logstash生成index也需要由原来的index => "filebeat-%{type}-%{+YYYY.MM.dd}"改为index => "filebeat-%{[fields][document_type]}-%{+YYYY.MM.dd}",最终index命名为filebeat-yewuname-*
4、最后和Elasic支持人员沟通后得知,客户端filebeat不升级也可以,不影响使用
二、Logstash
1、主要是自定义映射模板需要修改,模板这块儿改动比较大,可以先使用默认的获取格式后再修改,然后得出自己的模板运用,使用了geoip定位,相关类型需要修改,最终模板如下:cat /etc/logstash/templates/nginx_template { "template" : "filebeat-*", "settings" : { "index.refresh_interval" : "5s", "index.number_of_shards": "8", "index.number_of_replicas": "1" }, "mappings": { "_default_": { "properties": { "@timestamp": { "type": "date" }, "@version": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "agent": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "beat": { "type": "object" }, "clientRealIp": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "geoip": { "properties": { "city_name": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "continent_code": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "country_code2": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "country_code3": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "country_name": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "dma_code": { "type": "long" }, "ip": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "latitude": { "type": "float" }, "location": { "type": "geo_point" }, "longitude": { "type": "float" }, "postal_code": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "region_code": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "region_name": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "timezone": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "http_host": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "method": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "referrer": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "request_uri": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "responsetime": { "type": "float" }, "size": { "type": "long" }, "status": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "type": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "upstreamhost": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "url": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } } } }
2、输出到elasticsearch有些参数不支持了,需要注释flush_size,否则会报错The setting
flush_sizein plugin
elasticsearchis obsolete and is no longer available. This setting is no longer available as we now try to restrict bulk requests to sane sizes. See the 'Batch Sizes' section of the docs.
output { elasticsearch { hosts => ["192.168.88.240:9200","192.168.88.241:9200","192.168.88.242:9200","192.168.88.243:9200","192.168.88.244:9200"] index => "filebeat-%{[fields][document_type]}-%{+YYYY.MM.dd}" manage_template => true template_overwrite => true template_name => "nginx_template" template => "/etc/logstash/templates/nginx_template" #flush_size => 20000 #idle_flush_time => 5 } }
三、Elasticsearch
主要变更是该版本的一个Index只支持一个type了官方提供有滚动升级,按照这个步骤尝试了下没有成功,后删数据硬升级,如果大家想尝试,步骤如下
滚动升级
https://www.elastic.co/guide/en/elasticsearch/reference/5.6/rolling-upgrades.html
1、Disable shard allocation
例子:
curl -XPUT '192.168.88:9200/_cluster/settings?pretty' -H 'Content-Type: application/json' -d' { "transient": { "cluster.routing.allocation.enable": "none" } } '
例子:
curl -XPUT '192.168.88:9200/_cluster/settings?pretty' -H 'Content-Type: application/json' -d' > { > "transient": { > "cluster.routing.allocation.enable": "none" > } > } > ' { "acknowledged" : true, "persistent" : { }, "transient" : { "cluster" : { "routing" : { "allocation" : { "enable" : "none" } } } } }
2、Stop non-essential indexing and perform a synced flush (Optional)
curl -XPOST '192.168.88:9200/_flush/synced?pretty'
可选的,忽略
3、Stop and upgrade a single node
/etc/init.d/elasticsearch stop
rpm -e elasticsearch-5.2.2-1.noarch
rpm -ivh elasticsearch-6.0.0.rpm
chkconfig --add elasticsearch
chkconfig elasticsearch on
cp elasticsearch.yml.rpmsave elasticsearch.yml
注意权限,卸载rpm包再安装rpm包后elasticsearch的uid与gid会变化
chown elasticsearch:elasticsearch /data/eslog/ -R
chown elasticsearch:elasticsearch /data/esngx1/ -R
chown elasticsearch:elasticsearch /data/esngx2/ -R
4、Upgrade any plugins
没有第三方插件,忽略
5、Start the upgraded node
curl -XGET '192.168.88:9200/_cat/nodes?pretty'
6、Reenable shard allocation
curl -XPUT '192.168.88:9200/_cluster/settings?pretty' -H 'Content-Type: application/json' -d' { "transient": { "cluster.routing.allocation.enable": "all" } } '
7、Wait for the node to recover
curl -XGET '192.168.88:9200/_cat/health?pretty'
curl -XGET '192.168.88:9200/_cat/recovery?pretty'
8、Repeat
四、Kibana
1、index名字是字段type加通配2、导出老visualize模板后所有的visualize里面.raw改成.keyword,然后导入时dashboard需要与index一一对应
五、Grafana
同样修改数据源的index名字与查询语法.raw关键字为.keyword即可六、Elastalert
由于https://github.com/Yelp/elastalert/releases 安装部署是最新版本是v0.1.25还不支持elasticsearch6.0,后参考https://github.com/Yelp/elastalert/pull/1426 修改/root/elastalert-0.1.25/elastalert目录下elastalert.py、create_index.py、ruletypes.py、test_rule.py四个文件,最后安装(备注以后安装后可以直接复制该目录到其他服务器上
安装,不需要再修改源码)
现在v0.1.26版本更新就支持elasticsearch6.0了,直接更新新版本就行。
七、其他的kafka、nginx等不需要调整。
相关文章推荐
- centOS升级注意事项
- 大数据量Elastic数据迁移方法及注意事项
- 大数据量Elastic数据迁移方法及注意事项
- AW笔记本升级SSD,外接双屏中的一些注意事项
- 如何升级Ceph版本及注意事项
- STM32f030 boot iap升级时keil配置注意事项
- SQL Server 2008 R2 SP1升级到SQL Server 2012后的注意事项
- Web项目从Oracle转为Mysql,fluentnhibernate-1.0和NHibernate2.1.0升级到NHibernate3.3的注意事项
- WeX5 APP发布及版本升级注意事项
- U821升级到U810.1注意事项
- oracle数据库的升级注意事项
- gitlab 6.2 to 6.3升级注意事项
- R14版的ObjectARX程序向2002升级的注意事项
- 升级到php5.3要注意的事项
- windows下升级PHP到5.3.3的过程及注意事项
- Hive版本升级问题及注意事项
- 大数据量Elastic数据迁移方法及注意事项
- android系统级应用静默升级及注意事项
- DS4000安装升级实施过程中的重要注意事项
- VB6升级前的注意事项