Elasticsearch5.4集群（一）安装部署

转载自：http://blog.csdn.net/u013673976/article/details/73883617

ES版本升级

生产环境用的是1.7，Elasticsearch5.x在性能上有了很大的提升，计划升级到5.4，先在线下部署验证，过程中发现很多配置项都改了，各种报错。

一定要看官方文档：https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html，网上找到的很多都过时的低版本配置，要么不对要么不全，还需要注意下breaking changes(不兼容变更项)。

节点功能分离

master、data、client分开部署，规划存储容量、内存、CPU(权威指南：https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html)。master节点不存储数据，内存和cpu都比较低，一般部署3个master保证可靠。不建议用一块很大的磁盘，建议用多块SSD，可以并发IO。不是clien的节点可以不开启http服务的节点(http.enabled: false)同时也不要安装head, bigdesk, marvel等监控插件，这样保证data节点服务器只需处理创建/更新/删除/查询索引数据等操作。

内存分配

内存不要超过32G，java版本使用jdk1.8(We recommend installing Java version 1.8.0_131 or later)，es最大分配-Xms32766m -Xmx32766m（内存对象指针压缩技术），GC使用G1不要设置-Xmn(参照垃圾回收配置：http://www.oracle.com/technetwork/articles/java/g1gc-1984535.html)。物理机上的内存之分一半给ES，另一半给Lucene，否则性能会受影响。128G内存的机器最多部署2个es实例。

系统配置修改

禁用swap（在一些内核版本，swappness=0会引发OOM）。vim /etc/sysctl.conf

修改 vm.swappiness=1，设置vm.max_map_count=262144(否则启动会报错：max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144])

[plain] view
plain copy

vm.swappiness=1

vm.max_map_count=262144

立即生效需执行 sysctl -p

修改es用户限制，不能用root启动Elasticsearch，比如使用tomcat用户（用户重新登录生效）

vim /etc/security/limits.conf

[plain] view
plain copy

tomcat soft memlock unlimited

tomcat hard memlock unlimited


安装Elasticsearch5.4.1

安装步骤参照（https://www.elastic.co/downloads/elasticsearch），我们把es安装到/usr/local/目录下

[plain] view
plain copy

cd /usr/local/

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.4.1.tar.gz
sha1sum elasticsearch-5.4.1.tar.gz

tar -xzf elasticsearch-5.4.1.tar.gz

cd elasticsearch-5.4.1/

建立软链接:

[plain] view
plain copy

ln -s /usr/local/elasticsearch-5.4.1 elasticsearch

更新安装目录/usr/local/elasticsearch/bin/elasticsearch启动脚本，修改查找jvm配置文件顺序，加入"$LEIDA_ES_HOME"/config/jvm.options，让虚拟机参数优先使用实例自己的配置，因为我们一个物理机上会启用多个es实例

[plain] view
plain copy

if [ -z "$ES_JVM_OPTIONS" ]; then

for jvm_options in "$LEIDA_ES_HOME"/config/jvm.options \

"$ES_HOME"/config/jvm.options \

/etc/elasticsearch/jvm.options; do

if [ -r "$jvm_options" ]; then

ES_JVM_OPTIONS=$jvm_options

break

fi

done

fi

修改JVM参数的堆大小和G1GC方式(vim /usr/local/elasticsearch/config/jvm.options):

[plain] view
plain copy

-Xms32766m

-Xmx32766m

## GC configuration

#-XX:+UseConcMarkSweepGC

#-XX:CMSInitiatingOccupancyFraction=75

#-XX:+UseCMSInitiatingOccupancyOnly

## G1GC

-XX:+UseG1GC

-XX:MaxGCPauseMillis=800

-XX:ParallelGCThreads=15

-XX:ConcGCThreads=4


mmseg分词器

下载elasticsearch-analysis-mmseg-5.4.1.zip

下载地址：https://github.com/medcl/elasticsearch-analysis-mmseg/releases

(elasticsearch.yml不支持参数path.plugins参数，如果一台机器上启多个实例，想配置每个实例自己的插件，比如master节点不加载head插件，需要改下源码读取路径和check验证，默认是读取安装路径下的/usr/local/elasticsearch/plugins/)

[plain] view
plain copy

cd /usr/local/elasticsearch/plugins/

unzip elasticsearch-analysis-mmseg-5.4.1.zip -d elasticsearch-analysis-mmseg-5.4.1


创建master

实例配置存放在/data/es/es_[tcpport]下

[plain] view
plain copy

cd /data/es

mkdir -p es_9300/bin

mkdir -p es_9300/config

mkdir -p es_9300/data

mkdir -p es_9300/logs

cp /usr/local/elasticsearch/config/log4j2.properties /data/es/es_9300/config/log4j2.properties

cp /usr/local/elasticsearch/config/jvm.options /data/es/es_9300/config/jvm.options

修改JVM堆大小，vim /data/es/es_9300/config/jvm.options，安装目录的默认-Xms32766m是给datanode用的，master的堆需要改小。

cp /usr/local/elasticsearch/config/elasticsearch.yml /data/es/es_9300/config/elasticsearch.yml

修改elasticsearch.yml如下, IP、端口和存储路径需要自己替换，其他2个master的配置类似。为避免脑裂，discovery.zen.minimum_master_nodes设为2（非生产环境验证优先保证可用这里设为1）

[plain] view
plain copy

bootstrap.memory_lock: true

bootstrap.system_call_filter: false

cluster.name: loganalysis

node.attr.box_type: master

node.name: master_1

node.master: true

node.data: false

discovery.zen.minimum_master_nodes: 1

discovery.zen.ping.unicast.hosts: ["master1ip:9300","master2ip:9300","master3ip:9300"]

discovery.zen.fd.ping_timeout: 50s

discovery.zen.fd.ping_retries: 6

cluster.routing.allocation.node_initial_primaries_recoveries: 6

cluster.routing.allocation.node_concurrent_recoveries: 6

cluster.routing.allocation.cluster_concurrent_rebalance: 4

network.host: master1ip

http.port: 9200

transport.tcp.port: 9300

path.conf: /data/es/es_9300/config

path.data: /data/es/es_9300/data

path.logs: /data/es/es_9300/logs

http.cors.enabled: true

http.cors.allow-origin: /.*/

http.cors.allow-credentials: true


启动脚本

因为一个物理机上只安装了一个es，但是要启用多个es实例，需要区分config文件的路径，把别的脚本拷贝过来改成我们的leidaES.sh，启动master

[plain] view
plain copy

cd /data/es/es_9300/bin

./leidaES.sh start

elasticsearch's real path : /usr/local/elasticsearch-5.4.1

Starting elasticsearch es_9300: [ OK ]

脚本内容其实就是加了些验证和start、stop参数，停止命令./leidaES.sh stop

[plain] view
plain copy

#!/bin/sh

#

# init.d / servicectl compatibility (openSUSE)

#

if [ -f /etc/rc.status ]; then

. /etc/rc.status

rc_reset

fi

#

# Source function library.

#

if [ -f /etc/rc.d/init.d/functions ]; then

. /etc/rc.d/init.d/functions

fi

MAX_OPEN_FILES=600000

# max locked memory (kbytes, -l) 32G=1024*1024*32

MAX_LOCKED_MEMORY=33554432

# vim /etc/sysctl.conf add vm.max_map_count=262144

#MAX_MAP_COUNT=262144

ES_USER="tomcat"

ES_GROUP="tomcat"

JAVA_HOME=/usr/local/java/jdk1.8

ES_HOME="/usr/local/elasticsearch"

SCRIPT="$0"

# determine leida elasticsearch home

LEIDA_ES_HOME=`dirname "$SCRIPT"`/..

# make LEIDA ELASTICSEARCH_HOME absolute

LEIDA_ES_HOME=`cd "$LEIDA_ES_HOME"; pwd`

ES_NODE=`echo $LEIDA_ES_HOME|awk -F'/' '{print $NF}'`

CONF_DIR="${LEIDA_ES_HOME}/config"

WORK_DIR="/tmp/elasticsearch"

CONF_FILE="${CONF_DIR}/elasticsearch.yml"

#export ES_HOME

#export JAVA_HOME

export LEIDA_ES_HOME

exec="$ES_HOME/bin/elasticsearch"

if [ -f "$exec" ]; then

chmod 755 "$exec"

fi

prog="elasticsearch"

pidfile="$LEIDA_ES_HOME/${prog}.pid"

lockfile=/var/lock/subsys/$prog

# backwards compatibility for old config sysconfig files, pre 0.90.1

if [ -n $USER ] && [ -z $ES_USER ] ; then

ES_USER=$USER

fi

checkJava() {

if [ -x "$JAVA_HOME/bin/java" ]; then

JAVA="$JAVA_HOME/bin/java"

else

JAVA=`which java`

fi

if [ ! -x "$JAVA" ]; then

echo "Could not find any executable java binary. Please install java in your PATH or set JAVA_HOME"

exit 1

fi

}

start() {

checkJava

[ -x $exec ] || echo "$exec script file is not executable." ||exit 5

[ -f $CONF_FILE ] || echo "$CONF_FILE is not exists." || exit 6

# if [ -n "$MAX_LOCKED_MEMORY" -a -z "$ES_HEAP_SIZE" ]; then

# echo "MAX_LOCKED_MEMORY is set - ES_HEAP_SIZE must also be set"

# return 7

# fi

if [ -n "$MAX_OPEN_FILES" ]; then

ulimit -n $MAX_OPEN_FILES

fi

if [ -n "$MAX_LOCKED_MEMORY" ]; then

ulimit -l $MAX_LOCKED_MEMORY

fi

if [ -n "$MAX_MAP_COUNT" -a -f /proc/sys/vm/max_map_count ]; then

sysctl -q -w vm.max_map_count=$MAX_MAP_COUNT

fi

if [ -n "$WORK_DIR" ]; then

mkdir -p "$WORK_DIR"

chown "$ES_USER":"$ES_GROUP" "$WORK_DIR"

fi

# Ensure that the LEIDA_ES_HOME exists (it is cleaned at OS startup time)

if [ -n "$LEIDA_ES_HOME" ] && [ ! -e "$LEIDA_ES_HOME" ]; then

mkdir -p "$LEIDA_ES_HOME" && chown "$ES_USER":"$ES_GROUP" "$LEIDA_ES_HOME"

fi

chown -R "$ES_USER":"$ES_GROUP" "$LEIDA_ES_HOME"

chown -R "$ES_USER":"$ES_GROUP" "${ES_HOME}"

if [ -h "$ES_HOME" ]; then

eslink=`ls -ld "$ES_HOME"`

# Drop everything prior to ->

eslink=`expr "$eslink" : '.*-> .∗$'`

echo "elasticsearch's real path : $eslink"

if expr "$eslink" : '/.*' > /dev/null; then

chown -R "$ES_USER":"$ES_GROUP" "${eslink}"

fi

fi

if [ -n "$pidfile" ] && [ ! -e "$pidfile" ]; then

touch "$pidfile" && chown "$ES_USER":"$ES_GROUP" "$pidfile"

fi

echo -n $"Starting $prog ${ES_NODE}: "

# if not running, start it up here, usually something like "daemon $exec" # -Djava.io.tmpdir=${WORK_DIR}

daemon --user $ES_USER --pidfile $pidfile $exec -d -p $pidfile -Epath.conf=${CONF_DIR}

retval=$?

echo

[ $retval -eq 0 ] && touch $lockfile

return $retval

}

stop() {

echo -n $"Stopping $prog ${ES_NODE}: "

# stop it here, often "killproc $prog"

killproc -p $pidfile -d 20 $prog

retval=$?

echo

[ $retval -eq 0 ] && rm -f $lockfile

return $retval

}

restart() {

stop

start

}

reload() {

restart

}

force_reload() {

restart

}

rh_status() {

# run checks to determine if the service is running or use generic status

status -p $pidfile $prog

}

rh_status_q() {

rh_status >/dev/null 2>&1

}

case "$1" in

start)

rh_status_q && exit 0

$1

;;

stop)

rh_status_q || exit 0

$1

;;

restart)

$1

;;

reload)

rh_status_q || exit 7

$1

;;

force-reload)

force_reload

;;

status)

rh_status

;;

condrestart|try-restart)

rh_status_q || exit 0

restart

;;

*)

echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload}"

exit 2

esac

exit $?


创建datanode

和master节点的配置类似，注意node.master:false和node.data:true；node.name在集群中唯一。生产环境我们要做冷热分离，es1.7是通过分组来实现，ES5已改成node.attr.box_type。另外最好datanode和clientnode独立，非生产环境就合二为一了。cp /usr/local/elasticsearch/config/elasticsearch.yml /data/es/es_9301/config/elasticsearch.yml修改elasticsearch.yml如下：

[plain] view
plain copy

bootstrap.memory_lock: true

bootstrap.system_call_filter: false

cluster.name: loganalysis

node.attr.box_type: hot

node.name: hot_1

node.master: false

node.data: true

discovery.zen.minimum_master_nodes: 1

discovery.zen.ping.unicast.hosts: ["master1ip:9300","master2ip:9300","master3ip:9300"]

discovery.zen.fd.ping_timeout: 50s

discovery.zen.fd.ping_retries: 6

cluster.routing.allocation.node_initial_primaries_recoveries: 6

cluster.routing.allocation.node_concurrent_recoveries: 6

cluster.routing.allocation.cluster_concurrent_rebalance: 4

network.host: 改成机器的IP

http.port: 9201

transport.tcp.port: 9301

path.conf: /data/es/es_9301/config

path.data: /data/es/es_9301/data

path.logs: /data/es/es_9301/logs

http.cors.enabled: true

http.cors.allow-origin: /.*/

http.cors.allow-credentials: true


启动数据节点

先启动3个master节点，再启动配置的多个数据节点，组成Es集群

[plain] view
plain copy

cd /data/es/es_9301/bin

./leidaES.sh start

elasticsearch's real path : /usr/local/elasticsearch-5.4.1

Starting elasticsearch es_9301: [ OK ]


创建动态模版

配置mmseg分词和默认一个分片，不要副本。每个doc都有time字段和存储日志的msg字段。

[plain] view
plain copy

PUT /_template/default_template

{

"template": "*",

"settings": {

"index.refresh_interval": "15s",

"index.translog.flush_threshold_size": "1gb",

"index.translog.sync_interval": "1000000ms",

"index.codec": "best_compression",

"index.number_of_replicas": "0",

"index.number_of_shards": "1"

},

"mappings": {

"_default_": {

"_source": {

},

"dynamic_templates": [

{

"msg": {

"path_match": "msg",

"mapping": {

"analyzer": "mmseg_maxword",

"search_analyzer": "mmseg_maxword",

"omit_norms": true,

"type": "text"

}

}

},

{

"time": {

"path_match": "time",

"mapping": {

"type": "date"

}

}

},

{

"other": {

"mapping": {

"fielddata": {

"format": "doc_values"

},

"index": "not_analyzed",

"ignore_above":1000,

"omit_norms": true,

"doc_values": true

},

"match": "*"

}

}

],

"_all": {

"enabled": false

}

}

},

"aliases": {}

}

测试中文分词`mmseg_maxword` ,`mmseg_complex` ,`mmseg_simple`

[plain] view
plain copy

curl -X GET "localhost:9200/_analyze?analyzer=mmseg_maxword&pretty=true" -d '中华人民共和国12**3朝阳群众，www.elastic.co'


集群状态API接口测试

查看集群状态

curl 'localhost:9200/_cat/health?v'

创建索引（sense插件执行）

PUT /test

插入doc数据

POST /test/china/1

{"msg":"美国留给伊拉克的是个烂摊子吗"}

POST /test/china/2

{"msg":"公安部：各地校车将享最高路权"}

POST /test/china/3

{"msg":"中韩渔警冲突调查：韩警平均每天扣1艘中国渔船"}

POST /test/china/4

{"msg":"中国驻洛杉矶领事馆遭亚裔男子枪击 嫌犯已自首"}

搜索数据，能搜到2条数据。

POST /test/china/_search?

{"query":{"bool":{"must":[{"match_phrase_prefix":{"msg":{"query":"中国"}}}],"must_not":[]}},"from":0,"size":40}

至此集群正常运行，还需安装监控插件，后续优化配置