Zabbix监控tomcat连接超时排查和处理

近期由于业务项目的需要，考虑用zabbix jvm监控tomcat的一些性能等数据，中途出现了一些故障和问题，现记录下来分享给大家。
基础环境：

zabbix server： CentOS 7.2 IP：10.201.60.11 zabbix_java_getway zabbix 3.2

Tomcat server : CentOS 7.2 IP：10.201.30.15 tomcat 8 jdk 8

硬件防火墙：华为USG6306
10.201.60.11与10.201.30.15防火墙规则为 zabbix server 可以访问 Tomcat server的10050端口，
关于配置zabbix server与zabbix_java_getway等配置不做详细介绍，以下是相关的配置文件：

zabbix_java_getway.conf默认配置即可

在zabbix_server.conf里面添加以下信息
JavaGateway=10.201.60.11
StartJavaPollers=5
JavaGatewayPort=10052
启动相关服务。
更改tomcat主机的tomcat配置文件

[admin@bund-www-p-030015 bin]$ pwd
/usr/install/tomcat8/bin
[admin@bund-www-p-030015 bin]$ ls
bootstrap.jar       commons-daemon-native.tar.gz  digest.sh         startup.bat           tool-wrapper.sh
catalina.bat        configtest.bat                setclasspath.bat  startup.sh            version.bat
catalina.sh         configtest.sh                 setclasspath.sh   tomcat-juli.jar       version.sh
catalina-tasks.xml  daemon.sh                     shutdown.bat      tomcat-native.tar.gz
commons-daemon.jar  digest.bat                    shutdown.sh       tool-wrapper.bat
[admin@bund-www-p-030015 bin]$
在catalina.sh 文件中添加以下配置一般是加在配置文件前面。

CATALINA_OPTS="
-Dcom.sun.management.jmxremote
-Djavax.management.builder.initial=
-Djava.rmi.server.hostname=10.201.30.15
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.port=12345"
重启tomcat程序，用cmdline-jmxclient-0.10.3 做相关的测试  获取数据正常
[admin@bund-www-p-030015 bin]$ java   -jar   /tmp/cmdline-jmxclient-0.10.3.jar  - 10.201.30.15:12345 java.lang:type=Memory NonHeapMemoryUsage
08/25/2017 13:03:26 +0800 org.archive.jmx.Client NonHeapMemoryUsage:
committed: 82509824
init: 2555904
max: -1
used: 80213592

在zabbix_server上测试有以下报错，防火墙上面已经开启了12345端口的策略，但是链接过程中还是出现超时现象，用telnet测试12345端口，确实是可以访问的，zabbix的UI界面也是链接超时，于是查资料，说jvm在监控tomcat的时候，除了12345端口外还会随机开启两个端口，提供相关的服务，于是判断，这肯定是防火墙策略过高，于是将所有的端口开放，再检查竟然是可以连接的。

[admin@10-201-60-11 zabbix]$ java -jar /tmp/cmdline-jmxclient-0.10.3.jar  - 10.201.30.15:12345 java.lang:type=Memory NonHeapMemoryUsage
Exception in thread "main" java.rmi.ConnectException: Connection refused to host: 10.201.30.15; nested exception is:
java.net.ConnectException: Connection timed out (Connection timed out)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:130)
at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2430)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
at org.archive.jmx.Client.execute(Client.java:225)
at org.archive.jmx.Client.main(Client.java:154)
Caused by: java.net.ConnectException: Connection timed out (Connection timed out)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 9 more

更改防火墙后，访问正常，看来真是防火墙端口问题，这下可坑爹了，难道就将防火墙的端口开设成12345-65535吗？

[admin@10-201-60-11 zabbix]$ java -jar /tmp/cmdline-jmxclient-0.10.3.jar  - 10.201.30.15:12345 java.lang:type=Memory NonHeapMemoryUsage
08/25/2017 13:09:47 +0800 org.archive.jmx.Client NonHeapMemoryUsage:
committed: 82509824
init: 2555904
max: -1
used: 80266536

那这样的安全系数和策略太差劲了，所以继续度娘，找相关的配置，有类似指定端口类的说法，在tomcat的配置文件server.xml中添加以下配置。

<Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener"  rmiRegistryPortPlatform="12345" rmiServerPortPlatform="12346"/>

<Server port="8005" shutdown="SHUTDOWN">
<Listener className="org.apache.catalina.startup.VersionLoggerListener" />
<!-- Security listener. Documentation at /docs/config/listeners.html
<Listener className="org.apache.catalina.security.SecurityListener" />
-->
<!--APR library loader. Documentation at /docs/apr.html -->
<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
<!-- Prevent memory leaks due to use of particular java/javax APIs-->
<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
<Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
<Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
<Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener"  rmiRegistryPortPlatform="12345" rmiServerPortPlatform="12346"/>

重启tomcat服务发现类似12345端口已经绑定的错误，突然想起之前在catalina.sh文件中指定一个12345端口 在server.xml中又配置了12345端口，于是继续改，将catalina.sh里的12345端口注释掉，再启动tomcat，查看监听端口，

admin@bund-www-p-030015 conf]$ netstat  -nltup
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:10050           0.0.0.0:*               LISTEN      -
tcp6       0      0 :::8080                 :::*                    LISTEN      4926/java
tcp6       0      0 :::26613                :::*                    LISTEN      4926/java
tcp6       0      0 :::22                   :::*                    LISTEN      -
tcp6       0      0 :::12345                :::*                    LISTEN      4926/java
tcp6       0      0 :::12346                :::*                    LISTEN      4926/java
tcp6       0      0 :::10050                :::*                    LISTEN      -
tcp6       0      0 127.0.0.1:8005          :::*                    LISTEN      4926/java
tcp6       0      0 :::8009                 :::*                    LISTEN      4926/java
udp        0      0 0.0.0.0:40318           0.0.0.0:*                           -
udp        0      0 0.0.0.0:2583            0.0.0.0:*                           -
udp        0      0 0.0.0.0:63117           0.0.0.0:*                           -
udp        0      0 0.0.0.0:68              0.0.0.0:*                           -
udp        0      0 10.201.30.15:123        0.0.0.0:*                           -
udp        0      0 127.0.0.1:123           0.0.0.0:*                           -
udp        0      0 0.0.0.0:123             0.0.0.0:*                           -
udp6       0      0 :::37436                :::*                                -
udp6       0      0 fe80::f8db:66ff:fe6:123 :::*                                -
udp6       0      0 ::1:123                 :::*                                -
udp6       0      0 :::123

发现12345、12346端口处于监听，然后将防火墙策略改为12345-12346端口访问。再测试正常，基本问题解决。