问题描述

　　生产环境下有几台tomcat,但突然某个时候发现所有的请求都不能响应了,由于我们的web server使用的是nginx,会将请求反向到tomcat上,所以起初怀疑是nginx就没有收到请求,但查看日志后发现,nginx中大量出现499的返回,这说明问题还是出在tomcat上.

　　问题排查

　　首先我想到的是不是CPU跑满了,虽说CPU没有报警但还是本能的top命令看下系统负载,发现系统只有0.x的负载,cpu,内存消耗都是正常的.

　　由于CPU没有出现异常,所以应该不是GC出现了问题,但还是检查了下GC log,果然GC也没问题

　　此时必须让jstack上场了,果然在使用jstack后发现很多线程都是WAITING状态

"http-nio-127.0.0.1-801-exec-498" daemon prio=10 tid=0x00002ada7c14f800 nid=0x16a6 waiting on condition [0x00002ada9c905000]

　　java.lang.Thread.State: WAITING (parking)

　　at sun.misc.Unsafe.park(Native Method)

　　- parking to wait for <0x00000007873e6990> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

　　at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

　　at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

　　at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:133)

　　at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:282)

　　at org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64)

　　at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:177)

　　at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:170)

　　at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:102)

　　at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:240)

　　at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:227)

　　at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:173)

　　at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)

　　at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:85)

　　at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)

　　at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)

　　at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)

　　at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)

　　at com.weimai.utils.HttpClientUtil.doGet(HttpClientUtil.java:105)

　　at com.weimai.utils.HttpClientUtil.doGet(HttpClientUtil.java:87)

　　at com.weimai.utils.WeiBoUtil.checkUser(WeiBoUtil.java:214)

　　at com.weimai.web.UserInfoController.newWeiboLogin(UserInfoController.java:1223)

　　at sun.reflect.GeneratedMethodAccessor390.invoke(Unknown Source)

　　at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

　　at java.lang.reflect.Method.invoke(Method.java:606)