搭建ELK日志分析系统（一）-Elasticsearch安装

前言

系统介绍请参考

http://blog.csdn.net/uisoul/article/details/77741925

搭建ELK系统有两种方式

1、组件独立安装（更深入了解ELK系统的工作流程）

2、使用docker容器安装（这种方式配置更简单，快捷方便）

本系列文章使用组件独立安装的方式，如果你想使用docker容器安装，请跳过本教程

环境需求

本教程使用CentOS 6.9

Elasticsearch5.x版本不能以root用户运行，所以需增加普通用户

这里新增用户：elk

useradd elk

JDK环境安装

Elasticsearch是基于Java的，需要安装最新的JDK 1.8

如果自带低版本的JDK，执行以下命令查看

java -version

请删除再进行安装新版本

yum remove java

下载JDK1.8

wget http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz #wget命令可能下载不了，自行想办法下载

解压JDK1.8

#解压到指定目录
tar -zxvf ./jdk-8u144-linux-x64.tar.gz -C /usr/local/
#最终结果/usr/local/jdk1.8.0_144

配置环境变量

vi /etc/profile

新增

JAVA_HOME=/usr/local/jdk1.8.0_144
JRE_HOME=/usr/local/jdk1.8.0_144/jre
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
CLASSPATH=:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib

export PATH=$PATH:$JAVA_HOME:$JRE_HOME:$CLASSPATH

安装Elasticsearch5.5.2

下载安装包

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.2.tar.gz

解压安装包

#解压到指定目录
tar -zxvf ./elasticsearch-5.5.2.tar.gz -C /usr/local/
#最终结果/usr/local/elasticsearch-5.5.2
#改变权限
chown -R elk:elk /usr/local/elasticsearch-5.5.2/

直接运行elasticsearch

#切换用户
su elk
#进入目录
cd /usr/local/elasticsearch-5.5.2
#运行
./bin/elasticsearch

此时可能会出现一大堆异常

java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
at org.elasticsearch.bootstrap.SystemCallFilter.linuxImpl(SystemCallFilter.java:363) ~[elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.bootstrap.SystemCallFilter.init(SystemCallFilter.java:638) ~[elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.bootstrap.JNANatives.tryInstallSystemCallFilter(JNANatives.java:245) [elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.bootstrap.Natives.tryInstallSystemCallFilter(Natives.java:113) [elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:111) [elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:194) [elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:351) [elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) [elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.5.2.jar:5.5.2]
[2017-08-30T10:29:44,497][INFO ][o.e.n.Node               ] [] initializing ...
[2017-08-30T10:29:45,180][INFO ][o.e.e.NodeEnvironment    ] [2NJyxQe] using [1] data paths, mounts [[/ (/dev/mapper/VolGroup-lv_root)]], net usable_space [14.9gb], net total_space [17.1gb], spins? [possibly], types [ext4]
[2017-08-30T10:29:45,183][INFO ][o.e.e.NodeEnvironment    ] [2NJyxQe] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-08-30T10:29:45,197][INFO ][o.e.n.Node               ] node name [2NJyxQe] derived from node ID [2NJyxQewTSmbEGY0YlXTSg]; set [node.name] to override
[2017-08-30T10:29:45,201][INFO ][o.e.n.Node               ] version[5.5.2], pid[1828], build[b2f0c09/2017-08-14T12:33:14.154Z], OS[Linux/2.6.32-696.el6.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_144/25.144-b01]
[2017-08-30T10:29:45,205][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/local/elasticsearch-5.5.2]
[2017-08-30T10:29:48,468][INFO ][o.e.p.PluginsService     ] [2NJyxQe] loaded module [aggs-matrix-stats]
[2017-08-30T10:29:48,469][INFO ][o.e.p.PluginsService     ] [2NJyxQe] loaded module [ingest-common]
[2017-08-30T10:29:48,469][INFO ][o.e.p.PluginsService     ] [2NJyxQe] loaded module [lang-expression]
[2017-08-30T10:29:48,469][INFO ][o.e.p.PluginsService     ] [2NJyxQe] loaded module [lang-groovy]
[2017-08-30T10:29:48,469][INFO ][o.e.p.PluginsService     ] [2NJyxQe] loaded module [lang-mustache]
[2017-08-30T10:29:48,469][INFO ][o.e.p.PluginsService     ] [2NJyxQe] loaded module [lang-painless]
[2017-08-30T10:29:48,469][INFO ][o.e.p.PluginsService     ] [2NJyxQe] loaded module [parent-join]
[2017-08-30T10:29:48,469][INFO ][o.e.p.PluginsService     ] [2NJyxQe] loaded module [percolator]
[2017-08-30T10:29:48,469][INFO ][o.e.p.PluginsService     ] [2NJyxQe] loaded module [reindex]
[2017-08-30T10:29:48,469][INFO ][o.e.p.PluginsService     ] [2NJyxQe] loaded module [transport-netty3]
[2017-08-30T10:29:48,469][INFO ][o.e.p.PluginsService     ] [2NJyxQe] loaded module [transport-netty4]
[2017-08-30T10:29:48,470][INFO ][o.e.p.PluginsService     ] [2NJyxQe] no plugins loaded
[2017-08-30T10:29:55,362][INFO ][o.e.d.DiscoveryModule    ] [2NJyxQe] using discovery type [zen]
[2017-08-30T10:29:56,597][INFO ][o.e.n.Node               ] initialized
[2017-08-30T10:29:56,597][INFO ][o.e.n.Node               ] [2NJyxQe] starting ...
[2017-08-30T10:29:57,256][INFO ][o.e.t.TransportService   ] [2NJyxQe] publish_address {192.168.1.138:9300}, bound_addresses {192.168.1.138:9300}
[2017-08-30T10:29:57,296][INFO ][o.e.b.BootstrapChecks    ] [2NJyxQe] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-08-30T10:29:57,412][ERROR][o.e.b.Bootstrap          ] [2NJyxQe] node validation exception
[4] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[2]: max number of threads [1024] for user [elk] is too low, increase to at least [2048]
[3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[4]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2017-08-30T10:29:57,435][INFO ][o.e.n.Node               ] [2NJyxQe] stopping ...
[2017-08-30T10:29:57,534][INFO ][o.e.n.Node               ] [2NJyxQe] stopped
[2017-08-30T10:29:57,534][INFO ][o.e.n.Node               ] [2NJyxQe] closing ...
[2017-08-30T10:29:57,552][INFO ][o.e.n.Node               ] [2NJyxQe] closed

异常问题一

[4] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[2]: max number of threads [1024] for user [elk] is too low, increase to at least [2048]
[3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[4]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk

这里有4个问题，需要逐一解决

问题[1]

原因：无法创建本地文件问题,用户最大可创建文件数太小

解决方案：

切换到root用户，编辑limits.conf配置文件， 添加类似如下内容：

vi /etc/security/limits.conf

#添加如下内容:
*  soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096

问题[2]

解决：切换到root用户，修改90-nproc.conf配置文件。

vi /etc/security/limits.d/90-nproc.conf

#修改如下内容：
* soft nproc 1024
#修改为
* soft nproc 2048

问题[3]

解决：切换到root用户修改配置sysctl.conf

vi /etc/sysctl.conf
#添加下面配置：
vm.max_map_count=655360
#并执行命令：
sysctl -p

问题[4]

[4]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk

#和以下异常是统一原因
java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
....

问题原因：因为Centos6不支持SecComp，而ES5.5.2默认bootstrap.system_call_filter为true进行检测

所以导致检测失败，失败后直接导致ES不能启动。

详见 ：https://github.com/elastic/elasticsearch/issues/22899

解决方法：

修改配置

vi ./config/elasticsearch.yml

bootstrap.system_call_filter为false，注意要在Memory下面:

bootstrap.memory_lock: false
bootstrap.system_call_filter: false

切换root用户，重新启动elasticsearch

su root
./bin/elasticsearch

如无意外，启动正常

本机命令访问

curl http://127.0.0.1:9200

如出现以下内容，则启动正常

{
"name" : "2NJyxQe",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "HR3s74ntRXKZi0qal3BE0A",
"version" : {
"number" : "5.5.2",
"build_hash" : "b2f0c09",
"build_date" : "2017-08-14T12:33:14.154Z",
"build_snapshot" : false,
"lucene_version" : "6.6.0"
},
"tagline" : "You Know, for Search"
}

如果在虚拟机里安装elasticsearch，想使用外部浏览器访问

需修改配置

vi ./config/elasticsearch.yml
network.host: 192.168.1.138
http.port: 9200

# 增加新的参数，这样head插件可以访问es
http.cors.enabled: true
http.cors.allow-origin: "*"

重新启动elasticsearch，然后在window使用浏览器访问

http://192.168.1.138:9200/

如果访问出现问题，注意防火墙问题

关闭防火墙

service iptables stop

或防火墙配置中新增端口

su root
vi /etc/sysconfig/iptables
-A INPUT -m state --state NEW -m tcp -p tcp --dport 9200 -j ACCEPT
service iptables restart