记一次服务器CPU占用100%的故障排查经历

摘要: 线上有个web系统最近一直不稳定，不定时的服务器cpu就耗完，占用100%。重启tomcat后恢复正常，以后又会无规律的达到100%，满负荷运行。

线上有个web系统最近一直不稳定，不定时的服务器cpu就耗完，占用100%。重启tomcat后恢复正常，以后又会无规律的达到100%，满负荷运行。

首先想到的是有任务在后台一直运行或者代码有死循环，检查代码，关掉各种定时任务后更新服务器，还是会出现耗完cpu的现象。

于是用jstack把对应的tomcat进程的堆栈信息打印出来查看，发现业务线程执行很多步regex相关的代码。

$jstack -F 21609 > 1.log

打开1.log查看：

Thread 20212: (state = IN_JAVA)

- java.util.regex.Pattern$Curly.match0(java.util.regex.Matcher, int, int, java.lang.CharSequence) @bci=141, line=4170 (Compiled frame; information may be imprecise)

- java.util.regex.Pattern$Curly.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=54, line=4132 (Compiled frame)

- java.util.regex.Pattern$GroupHead.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=28, line=4556 (Compiled frame)

- java.util.regex.Pattern$Loop.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=106, line=4683 (Compiled frame)

- java.util.regex.Pattern$GroupTail.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=70, line=4615 (Compiled frame)

- java.util.regex.Pattern$Ques.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=68, line=4079 (Compiled frame)

- java.util.regex.Pattern$Curly.match0(java.util.regex.Matcher, int, int, java.lang.CharSequence) @bci=141, line=4170 (Compiled frame)

- java.util.regex.Pattern$Curly.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=54, line=4132 (Compiled frame)

- java.util.regex.Pattern$GroupHead.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=28, line=4556 (Compiled frame)

- java.util.regex.Pattern$Loop.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=106, line=4683 (Compiled frame)

- java.util.regex.Pattern$GroupTail.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=70, line=4615 (Compiled frame)

- java.util.regex.Pattern$Ques.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=68, line=4079 (Compiled frame)

- java.util.regex.Pattern$Curly.match0(java.util.regex.Matcher, int, int, java.lang.CharSequence) @bci=141, line=4170 (Compiled frame)

- java.util.regex.Pattern$Curly.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=54, line=4132 (Compiled frame)

- java.util.regex.Pattern$GroupHead.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=28, line=4556 (Compiled frame)

- java.util.regex.Pattern$Loop.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=106, line=4683 (Compiled frame)

- java.util.regex.Pattern$GroupTail.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=70, line=4615 (Compiled frame)

- java.util.regex.Pattern$Ques.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=68, line=4079 (Compiled frame)

- java.util.regex.Pattern$Curly.match0(java.util.regex.Matcher, int, int, java.lang.CharSequence) @bci=141, line=4170 (Compiled frame)

- java.util.regex.Pattern$Curly.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=54, line=4132 (Compiled frame)

- java.util.regex.Pattern$GroupHead.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=28, line=4556 (Compiled frame)

- java.util.regex.Pattern$Loop.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=106, line=4683 (Compiled frame)

- java.util.regex.Pattern$GroupTail.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=70, line=4615 (Compiled frame)

- java.util.regex.Pattern$Ques.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=68, line=4079 (Compiled frame)

- java.util.regex.Pattern$Curly.match0(java.util.regex.Matcher, int, int, java.lang.CharSequence) @bci=141, line=4170 (Compiled frame)

- java.util.regex.Pattern$Curly.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=54, line=4132 (Compiled frame)

- java.util.regex.Pattern$GroupHead.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=28, line=4556 (Compiled frame)

- java.util.regex.Pattern$Loop.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=106, line=4683 (Compiled frame)

- java.util.regex.Pattern$GroupTail.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=70, line=4615 (Compiled frame)

- java.util.regex.Pattern$Ques.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=68, line=4079 (Compiled frame)

- java.util.regex.Pattern$Curly.match0(java.util.regex.Matcher, int, int, java.lang.CharSequence) @bci=141, line=4170 (Compiled frame)

- java.util.regex.Pattern$Curly.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=54, line=4132 (Compiled frame)

- java.util.regex.Pattern$GroupHead.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=28, line=4556 (Compiled frame)

- java.util.regex.Pattern$Loop.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=106, line=4683 (Compiled frame)

- java.util.regex.Pattern$GroupTail.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=70, line=4615 (Compiled frame)

- java.util.regex.Pattern$Ques.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=68, line=4079 (Compiled frame)

- java.util.regex.Pattern$Curly.match0(java.util.regex.Matcher, int, int, java.lang.CharSequence) @bci=141, line=4170 (Compiled frame)

- java.util.regex.Pattern$Curly.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=54, line=4132 (Compiled frame)

- java.util.regex.Pattern$GroupHead.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=28, line=4556 (Compiled frame)

- java.util.regex.Pattern$Loop.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=106, line=4683 (Compiled frame)

- java.util.regex.Pattern$GroupTail.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=70, line=4615 (Compiled frame)

- java.util.regex.Pattern$Ques.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=68, line=4079 (Compiled frame)

- java.util.regex.Pattern$Curly.match0(java.util.regex.Matcher, int, int, java.lang.CharSequence) @bci=141, line=4170 (Compiled frame)

- java.util.regex.Pattern$Curly.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=54, line=4132 (Compiled frame)

- java.util.regex.Pattern$GroupHead.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=28, line=4556 (Compiled frame)

- java.util.regex.Pattern$Loop.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=106, line=4683 (Compiled frame)

- java.util.regex.Pattern$GroupTail.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=70, line=4615 (Compiled frame)

- java.util.regex.Pattern$Ques.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=68, line=4079 (Compiled frame)

- java.util.regex.Pattern$Curly.match0(java.util.regex.Matcher, int, int, java.lang.CharSequence) @bci=141, line=4170 (Compiled frame)

- java.util.regex.Pattern$Curly.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=54, line=4132 (Compiled frame)

- java.util.regex.Pattern$GroupHead.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=28, line=4556 (Compiled frame)

- java.util.regex.Pattern$Loop.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=106, line=4683 (Compiled frame)

- java.util.regex.Pattern$GroupTail.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=70, line=4615 (Compiled frame)

- java.util.regex.Pattern$Ques.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=68, line=4079 (Compiled frame)

- java.util.regex.Pattern$Curly.match0(java.util.regex.Matcher, int, int, java.lang.CharSequence) @bci=141, line=4170 (Compiled frame)

- java.util.regex.Pattern$Curly.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=54, line=4132 (Compiled frame)

- java.util.regex.Pattern$GroupHead.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=28, line=4556 (Compiled frame)

- java.util.regex.Pattern$Loop.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=106, line=4683 (Compiled frame)

- java.util.regex.Pattern$GroupTail.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=70, line=4615 (Compiled frame)

- java.util.regex.Pattern$Ques.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=68, line=4079 (Compiled frame)

- java.util.regex.Pattern$Curly.match0(java.util.regex.Matcher, int, int, java.lang.CharSequence) @bci=141, line=4170 (Compiled frame)

- java.util.regex.Pattern$Curly.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=54, line=4132 (Compiled frame)

- java.util.regex.Pattern$GroupHead.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=28, line=4556 (Compiled frame)

- java.util.regex.Pattern$Loop.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=106, line=4683 (Compiled frame)

- java.util.regex.Pattern$GroupTail.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=70, line=4615 (Compiled frame)

- java.util.regex.Pattern$Ques.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=68, line=4079 (Compiled frame)

- java.util.regex.Pattern$Curly.match0(java.util.regex.Matcher, int, int, java.lang.CharSequence) @bci=141, line=4170 (Compiled frame)

- java.util.regex.Pattern$Curly.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=54, line=4132 (Compiled frame)

- java.util.regex.Pattern$GroupHead.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=28, line=4556 (Compiled frame)

- java.util.regex.Pattern$Loop.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=106, line=4683 (Compiled frame)

- java.util.regex.Pattern$GroupTail.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=70, line=4615 (Compiled frame)

- java.util.regex.Pattern$Ques.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=68, line=4079 (Compiled frame)

- java.util.regex.Pattern$Curly.match0(java.util.regex.Matcher, int, int, java.lang.CharSequence) @bci=141, line=4170 (Compiled frame)

- java.util.regex.Pattern$Curly.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=54, line=4132 (Compiled frame)

- java.util.regex.Pattern$GroupHead.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=28, line=4556 (Compiled frame)

- java.util.regex.Pattern$Loop.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=106, line=4683 (Compiled frame)

- java.util.regex.Pattern$GroupTail.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=70, line=4615 (Compiled frame)

- java.util.regex.Pattern$Ques.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=68, line=4079 (Compiled frame)

- java.util.regex.Pattern$Curly.match0(java.util.regex.Matcher, int, int, java.lang.CharSequence) @bci=141, line=4170 (Compiled frame)

- java.util.regex.Pattern$Curly.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=54, line=4132 (Compiled frame)

- java.util.regex.Pattern$GroupHead.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=28, line=4556 (Compiled frame)

- java.util.regex.Pattern$Loop.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=106, line=4683 (Compiled frame)

- java.util.regex.Pattern$GroupTail.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=70, line=4615 (Compiled frame)

- java.util.regex.Pattern$Ques.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=68, line=4079 (Compiled frame)

- java.util.regex.Pattern$Curly.match0(java.util.regex.Matcher, int, int, java.lang.CharSequence) @bci=141, line=4170 (Compiled frame)

- java.util.regex.Pattern$Curly.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=54, line=4132 (Compiled frame)

- java.util.regex.Pattern$GroupHead.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=28, line=4556 (Compiled frame)

- java.util.regex.Pattern$Loop.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=106, line=4683 (Compiled frame)

- java.util.regex.Pattern$GroupTail.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=70, line=4615 (Compiled frame)

- java.util.regex.Pattern$Ques.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=68, line=4079 (Compiled frame)

- java.util.regex.Pattern$Curly.match0(java.util.regex.Matcher, int, int, java.lang.CharSequence) @bci=141, line=4170 (Compiled frame)

- java.util.regex.Pattern$Curly.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=54, line=4132 (Compiled frame)

- java.util.regex.Pattern$GroupHead.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=28, line=4556 (Compiled frame)

- java.util.regex.Pattern$Loop.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=106, line=4683 (Compiled frame)

- java.util.regex.Pattern$GroupTail.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=70, line=4615 (Compiled frame)

- java.util.regex.Pattern$Ques.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=68, line=4079 (Compiled frame)

- java.util.regex.Pattern$Curly.match0(java.util.regex.Matcher, int, int, java.lang.CharSequence) @bci=141, line=4170 (Compiled frame)

- java.util.regex.Pattern$Curly.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=54, line=4132 (Compiled frame)

- java.util.regex.Pattern$GroupHead.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=28, line=4556 (Compiled frame)

- java.util.regex.Pattern$Loop.matchInit(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=39, line=4699 (Interpreted frame)

- java.util.regex.Pattern$Prolog.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=7, line=4639 (Interpreted frame)

- java.util.regex.Pattern$Begin.match(java.util.regex.Matcher, int, java.lang.CharSequence) @bci=30, line=3472 (Compiled frame)

- java.util.regex.Matcher.search(int) @bci=86, line=1199 (Compiled frame)

- java.util.regex.Matcher.find() @bci=65, line=592 (Interpreted frame)

- com.zte.appunistore.agent.dao.operationability.TradeAppDAO.isnotnull_email(java.lang.String) @bci=32, line=100 (Interpreted frame)

- com.zte.appunistore.agent.dao.operationability.TradeAppDAO.getAppDownUrl(java.lang.String, java.lang.String, java.lang.String, java.lang.String, java.lang.String, java.lang.String, int, java.lang.String[], java.lang.String) @bci=37, line=266 (Interpreted frame)

- com.zte.appunistore.agent.business.operationability.RedirectDownloadBusiness.execute() @bci=293, line=85 (Interpreted frame)

- com.zte.appunistore.agent.servlet.AppActionServlet.doGet(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=150, line=95 (Interpreted frame)

- javax.servlet.http.HttpServlet.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=35, line=622 (Interpreted frame)

- javax.servlet.http.HttpServlet.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=30, line=729 (Interpreted frame)

- org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=396, line=291 (Interpreted frame)

- org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=101, line=206 (Interpreted frame)

- org.apache.tomcat.websocket.server.WsFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=21, line=52 (Interpreted frame)

检查标红地方的代码，TradeAppDAO类的isnotnull_email方法，发现该方法是一个校验email是否符合格式规范的正则表达式匹配方法：

String checkPattern = "^([a-z0-9A-Z]+[-|\\.]?)+[a-z0-9A-Z]@([a-z0-9A-Z]+(-[a-z0-9A-Z]+)?\\.)+[a-zA-Z]{2,}$";

Pattern regex = Pattern.compile(checkPattern);

Matcher matcher = regex.matcher(email);

排查服务器请求日志，找到出现超时的请求：

GET /unistore/servicedata.do?&serviceid=redirectDownload&email=UUID09ebf8d83b3a4d68a64dbd93a514e219&productid=851c703d9ec440cab71d4faab7164b55&istry=1&packagename=com.xiaoao.card HTTP/1.0

带入请求email参数验证，发现传入“UUID09ebf8d83b3a4d68a64dbd93a514e219”字符串时，将本机cpu迅速耗完。到此，定位到了问题，应该是该正则表达式在匹配一些特定字符串时会出现匹配不到循环递归匹配的状况，将服务器cpu耗完。

修改正则表达式解决问题：

String checkPattern = "\\w+([-+.]\\w+)*@\\w+([-.]\\w+)*\\.\\w+([-.]\\w+)*";

看来正则表达式是把双刃剑，使用时要格外小心注意。