CPU占用100%排查过程

在一次项目过程中，cpu使用率一直处于接近100%中，到处查询资料，总结如下

排查过程

1. top

通过top命令找到，消耗CPU最高的PID,从下图发现PID为15913

2. top -p 15913 -H

加上-H 选项可以看到该进程的相关线程信息，从下图中可知最耗CPU的两个PID分别是15924和15925，对应的16进制为0x3E34和0x3E35

3. jstack 15913 > jstack.log

jstack命令可以得到线程堆栈信息，根据这些线程堆栈信息，可以去查看java程序出现的问题，如检测死。jstack.log的完整内容在文章最后发出来。

在日志的最后面找到0x3e34 和 0x3e35,对应的是GC线程，由此猜想可能是不停地GC导致CPU占用过高。

需要注意的是，在单核机器上，根本看不到GC线程，这个16进制数指向的是JVM Thread,试了好几次都是这样，一开始在这上面浪费太多时间了。

jstack 15913 | grep -A 10 0x3e34

4. jstat -gcutil 15193 1000 20

以上命令代码每过一秒打印一次gc情况，打印20次。

参数项说明如下：

由上图可知，20s内发生了13次Full GC,所以问题就是Full Gc 太频繁导致的，内存使用率太高，猜测可能是因为jvm分配的内存太小了。

5. jmap -heap 15193

以上命令用于查看java进程的JVM内存情况

最大内存为948M,老年代632M，使用率已经无线接近100%。机器有4G内存，但为该进程分配的JVM最大内存只有948M.

6. jmap -dump:format=b,file==heap.hprof 15193

以上命令用于导出jvm内存信息，这个我暂时也没看出什么。

7. 增大堆内存

-Xms2048m -Xmx2048m ,增大堆内存到2048M，重启查看效果。

说明：系统启动之后3s内，CPU占用还是100%，接下来就降下来了。 通过jstat查看GC情况，发生了16次YGC，发生了12次 FGC，难道是系统启动的时候，创建了大量的对象？

不过接下来系统运行平稳，先观察一段时间看看。

jstack日志

2019-02-25 16:16:48Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.73-b02 mixed mode):"Attach Listener" #133 daemon prio=9 os_prio=0 tid=0x00007f023000e000 nid=0x57b9 waiting on condition <0x0000000000000000>   java.lang.Thread.State: RUNNABLE"http-nio-8080-exec-12" #132 daemon prio=5 os_prio=0 tid=0x00007f0220018800 nid=0x5419 waiting on condition <0x00007f020fefd000>   java.lang.Thread.State: WAITING (parking)    at sun.misc.Unsafe.park(Native Method)    - parking to wait for  <0x00000000c79b3f10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)    at java.lang.Thread.run(Thread.java:745)"http-nio-8080-exec-11" #130 daemon prio=5 os_prio=0 tid=0x00007f022c62a800 nid=0x4dcd waiting on condition <0x00007f020f4d9000>   java.lang.Thread.State: WAITING (parking)    at sun.misc.Unsafe.park(Native Method)    - parking to wait for  <0x00000000c79b3f10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)    at java.lang.Thread.run(Thread.java:745)"Curator-TreeCache-5" #53 daemon prio=5 os_prio=0 tid=0x00007f022008e800 nid=0x3eb9 waiting on condition <0x00007f020c89c000>   java.lang.Thread.State: WAITING (parking)    at sun.misc.Unsafe.park(Native Method)    - parking to wait for  <0x00000000c83319a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)    at java.lang.Thread.run(Thread.java:745)"Curator-TreeCache-4" #52 daemon prio=5 os_prio=0 tid=0x00007f02108ab000 nid=0x3eac waiting on condition <0x00007f020c99d000>   java.lang.Thread.State: WAITING (parking)    at sun.misc.Unsafe.park(Native Method)    - parking to wait for  <0x00000000c99900c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)    at java.lang.Thread.run(Thread.java:745)"Curator-TreeCache-3" #51 daemon prio=5 os_prio=0 tid=0x00007f0240011000 nid=0x3eab waiting on condition <0x00007f020cc9e000>   java.lang.Thread.State: WAITING (parking)    at sun.misc.Unsafe.park(Native Method)    - parking to wait for  <0x00000000c615a470> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)    at java.lang.Thread.run(Thread.java:745)"Curator-TreeCache-2" #49 daemon prio=5 os_prio=0 tid=0x00007f02100c0000 nid=0x3ea9 waiting on condition <0x00007f020cea0000>   java.lang.Thread.State: WAITING (parking)    at sun.misc.Unsafe.park(Native Method)    - parking to wait for  <0x00000000c7c0daa8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)    at java.lang.Thread.run(Thread.java:745)"Curator-TreeCache-1" #48 daemon prio=5 os_prio=0 tid=0x00007f0240001800 nid=0x3ea8 waiting on condition <0x00007f020d801000>   java.lang.Thread.State: WAITING (parking)    at sun.misc.Unsafe.park(Native Method)    - parking to wait for  <0x00000000c7fa0a68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)    at java.lang.Thread.run(Thread.java:745)"Curator-TreeCache-0" #47 daemon prio=5 os_prio=0 tid=0x00007f0220085000 nid=0x3ea6 waiting on condition <0x00007f020d902000>   java.lang.Thread.State: WAITING (parking)    at sun.misc.Unsafe.park(Native Method)    - parking to wait for  <0x00000000c9991620> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)    at java.lang.Thread.run(Thread.java:745)"DestroyJavaVM" #46 prio=5 os_prio=0 tid=0x00007f0264008800 nid=0x3e31 waiting on condition <0x0000000000000000>   java.lang.Thread.State: RUNNABLE"Curator-Framework-0" #45 daemon prio=5 os_prio=0 tid=0x00007f026560c000 nid=0x3ea0 waiting on condition <0x00007f020da03000>   java.lang.Thread.State: WAITING (parking)    at sun.misc.Unsafe.park(Native Method)    - parking to wait for  <0x00000000c63cfc68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)    at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)    at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)    at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:804)    at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)    at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)    at java.util.concurrent.FutureTask.run(FutureTask.java:266)    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)    at java.lang.Thread.run(Thread.java:745)"main-EventThread" #44 daemon prio=5 os_prio=0 tid=0x00007f0265485800 nid=0x3e9f waiting on condition <0x00007f020db04000>   java.lang.Thread.State: WAITING (parking)    at sun.misc.Unsafe.park(Native Method)    - parking to wait for  <0x00000000c63cf5d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)"Curator-ConnectionStateManager-0" #42 daemon prio=5 os_prio=0 tid=0x00007f026515c800 nid=0x3e9d waiting on condition <0x00007f020dd06000>   java.lang.Thread.State: WAITING (parking)    at sun.misc.Unsafe.park(Native Method)    - parking to wait for  <0x00000000c63d0240> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)    at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)    at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:245)    at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)    at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:111)    at java.util.concurrent.FutureTask.run(FutureTask.java:266)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)    at java.lang.Thread.run(Thread.java:745)"Curator-Framework-0" #41 daemon prio=5 os_prio=0 tid=0x00007f02642db800 nid=0x3e9c waiting on condition <0x00007f020de07000>   java.lang.Thread.State: WAITING (parking)    at sun.misc.Unsafe.park(Native Method)    - parking to wait for  <0x00000000c8686310> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)    at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)    at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)    at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:804)    at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)    at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)    at java.util.concurrent.FutureTask.run(FutureTask.java:266)    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)    at java.lang.Thread.run(Thread.java:745)"main-EventThread" #40 daemon prio=5 os_prio=0 tid=0x00007f02646e3800 nid=0x3e9b waiting on condition <0x00007f020df08000>   java.lang.Thread.State: WAITING (parking)    at sun.misc.Unsafe.park(Native Method)    - parking to wait for  <0x00000000c8685c20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)"Curator-ConnectionStateManager-0" #38 daemon prio=5 os_prio=0 tid=0x00007f0264d01000 nid=0x3e99 waiting on condition <0x00007f020f1d6000>   java.lang.Thread.State: WAITING (parking)    at sun.misc.Unsafe.park(Native Method)    - parking to wait for  <0x00000000c86868e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)    at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)    at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:245)    at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)    at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:111)    at java.util.concurrent.FutureTask.run(FutureTask.java:266)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)    at java.lang.Thread.run(Thread.java:745)"Curator-Framework-0" #37 daemon prio=5 os_prio=0 tid=0x00007f02649f7800 nid=0x3e98 waiting on condition <0x00007f020f2d7000>   java.lang.Thread.State: WAITING (parking)    at sun.misc.Unsafe.park(Native Method)    - parking to wait for  <0x00000000c65c0318> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)    at java.lang.Thread.run(Thread.java:745)"Curator-ConnectionStateManager-0" #34 daemon prio=5 os_prio=0 tid=0x00007f0265228800 nid=0x3e95 waiting on condition <0x00007f020f9fa000>   java.lang.Thread.State: WAITING (parking)    at sun.misc.Unsafe.park(Native Method)    - parking to wait for  <0x00000000c65c08a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)    at java.lang.Thread.run(Thread.java:745)"http-nio-8080-Acceptor-0" #31 daemon prio=5 os_prio=0 tid=0x00007f0265b78000 nid=0x3e93 runnable <0x00007f020fdfc000>   java.lang.Thread.State: RUNNABLE    at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)    - locked <0x00000000c79c06b8> (a java.lang.Object)    at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:482)    at java.lang.Thread.run(Thread.java:745)"http-nio-8080-exec-10" #28 daemon prio=5 os_prio=0 tid=0x00007f0265b52000 nid=0x3e90 waiting on condition <0x00007f02341c8000>   java.lang.Thread.State: WAITING (parking)    at sun.misc.Unsafe.park(Native Method)    - parking to wait for  <0x00000000c79b3f10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)    at java.lang.Thread.run(Thread.java:745)"http-nio-8080-exec-9" #27 daemon prio=5 os_prio=0 tid=0x00007f0265b50000 nid=0x3e8f waiting on condition <0x00007f02342c9000>   java.lang.Thread.State: WAITING (parking)    at sun.misc.Unsafe.park(Native Method)    - parking to wait for  <0x00000000c79b3f10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)    at java.lang.Thread.run(Thread.java:745)"http-nio-8080-exec-8" #26 daemon prio=5 os_prio=0 tid=0x00007f0265b4d800 nid=0x3e8e waiting on condition <0x00007f02343ca000>   java.lang.Thread.State: WAITING (parking)    at sun.misc.Unsafe.park(Native Method)    - parking to wait for  <0x00000000c79b3f10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)    at java.lang.Thread.run(Thread.java:745)"http-nio-8080-exec-7" #25 daemon prio=5 os_prio=0 tid=0x00007f0265b4b800 nid=0x3e8d waiting on condition <0x00007f02344cb000>   java.lang.Thread.State: WAITING (parking)    at sun.misc.Unsafe.park(Native Method)    - parking to wait for  <0x00000000c79b3f10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)    at java.lang.Thread.run(Thread.java:745)"http-nio-8080-exec-4" #22 daemon prio=5 os_prio=0 tid=0x00007f0265b45800 nid=0x3e8a waiting on condition <0x00007f02347ce000>   java.lang.Thread.State: WAITING (parking)    at sun.misc.Unsafe.park(Native Method)    - parking to wait for  <0x00000000c79b3f10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)    at java.lang.Thread.run(Thread.java:745)"http-nio-8080-exec-3" #21 daemon prio=5 os_prio=0 tid=0x00007f0265b44000 nid=0x3e89 waiting on condition <0x00007f02348cf000>   java.lang.Thread.State: WAITING (parking)    at sun.misc.Unsafe.park(Native Method)    - parking to wait for  <0x00000000c79b3f10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)    at java.lang.Thread.run(Thread.java:745)"http-nio-8080-exec-2" #20 daemon prio=5 os_prio=0 tid=0x00007f0265b42000 nid=0x3e88 waiting on condition <0x00007f02349d0000>   java.lang.Thread.State: WAITING (parking)    at sun.misc.Unsafe.park(Native Method)    - parking to wait for  <0x00000000c79b3f10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:103)    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:31)    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)    at java.lang.Thread.run(Thread.java:745)"http-nio-8080-exec-1" #19 daemon prio=5 os_prio=0 tid=0x00007f0265b40800 nid=0x3e87 waiting on condition <0x00007f0234ace000>   java.lang.Thread.State: RUNNABLE    at sun.net.www.ParseUtil.encodePath(ParseUtil.java:105)    at sun.misc.URLClassPath$Loader.getResource(URLClassPath.java:692)    at sun.misc.URLClassPath.getResource(URLClassPath.java:212)    at java.net.URLClassLoader$1.run(URLClassLoader.java:365)    at java.net.URLClassLoader$1.run(URLClassLoader.java:362)    at java.security.AccessController.doPrivileged(Native Method)    at java.net.URLClassLoader.findClass(URLClassLoader.java:361)    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)    - locked <0x00000000c51cfe20> (a java.lang.Object)    at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:94)    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)    at org.springframework.util.ClassUtils.forName(ClassUtils.java:251)    at org.springframework.util.ClassUtils.isPresent(ClassUtils.java:306)    at org.springframework.boot.autoconfigure.thymeleaf.ThymeleafTemplateAvailabilityProvider.isTemplateAvailable(ThymeleafTemplateAvailabilityProvider.java:39)    at org.springframework.boot.autoconfigure.template.TemplateAvailabilityProviders.findProvider(TemplateAvailabilityProviders.java:159)    at org.springframework.boot.autoconfigure.template.TemplateAvailabilityProviders.getProvider(TemplateAvailabilityProviders.java:146)    - locked <0x00000000c56f2c38> (a org.springframework.boot.autoconfigure.template.TemplateAvailabilityProviders$1)    at org.springframework.boot.autoconfigure.template.TemplateAvailabilityProviders.getProvider(TemplateAvailabilityProviders.java:119)    at org.springframework.boot.autoconfigure.web.DefaultErrorViewResolver.resolve(DefaultErrorViewResolver.java:114)    at org.springframework.boot.autoconfigure.web.DefaultErrorViewResolver.resolveErrorView(DefaultErrorViewResolver.java:104)    at org.springframework.boot.autoconfigure.web.AbstractErrorController.resolveErrorView(AbstractErrorController.java:113)    at org.springframework.boot.autoconfigure.web.BasicErrorController.errorHtml(BasicErrorController.java:92)    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)    at java.lang.reflect.Method.invoke(Method.java:497)    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:849)    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:760)    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)    at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)    at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)    at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:728)    at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:472)    at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:395)    at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:316)    at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:395)    at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:254)    at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:349)    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:175)    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)    - locked <0x00000000ec3e3f18> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)    at java.lang.Thread.run(Thread.java:745)"NioBlockingSelector.BlockPoller-1" #18 daemon prio=5 os_prio=0 tid=0x00007f0265b10800 nid=0x3e86 runnable <0x00007f024d436000>   java.lang.Thread.State: RUNNABLE    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)    - locked <0x00000000c79cef18> (a sun.nio.ch.Util$2)    - locked <0x00000000c79cef08> (a java.util.Collections$UnmodifiableSet)    - locked <0x00000000c79cecd0> (a sun.nio.ch.EPollSelectorImpl)    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)    at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:298)"commons-pool-evictor-thread" #17 prio=5 os_prio=0 tid=0x00007f02659b8800 nid=0x3e85 waiting on condition <0x00007f0234bd2000>   java.lang.Thread.State: TIMED_WAITING (parking)    at sun.misc.Unsafe.park(Native Method)    - parking to wait for  <0x00000000c7c5dfa0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)    at java.lang.Thread.run(Thread.java:745)"container-0" #15 prio=5 os_prio=0 tid=0x00007f02644b5000 nid=0x3e77 waiting on condition <0x00007f02352d3000>   java.lang.Thread.State: TIMED_WAITING (sleeping)    at java.lang.Thread.sleep(Native Method)    at org.apache.catalina.core.StandardServer.await(StandardServer.java:427)    at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer$2.run(TomcatEmbeddedServletContainer.java:192)"Service Thread" #7 daemon prio=9 os_prio=0 tid=0x00007f0264152000 nid=0x3e3c runnable <0x0000000000000000>   java.lang.Thread.State: RUNNABLE"C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f026413d000 nid=0x3e3b waiting on condition <0x0000000000000000>   java.lang.Thread.State: RUNNABLE"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f026413a000 nid=0x3e3a waiting on condition <0x0000000000000000>   java.lang.Thread.State: RUNNABLE"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f0264138800 nid=0x3e39 runnable <0x0000000000000000>   java.lang.Thread.State: RUNNABLE"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f0264101000 nid=0x3e38 in Object.wait() <0x00007f024e05c000>   java.lang.Thread.State: WAITING (on object monitor)    at java.lang.Object.wait(Native Method)    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)    - locked <0x00000000c4c6a808> (a java.lang.ref.ReferenceQueue$Lock)    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f02640fe800 nid=0x3e37 in Object.wait() <0x00007f024e15d000>   java.lang.Thread.State: WAITING (on object monitor)    at java.lang.Object.wait(Native Method)    at java.lang.Object.wait(Object.java:502)    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)    - locked <0x00000000c4c6a848> (a java.lang.ref.Reference$Lock)"VM Thread" os_prio=0 tid=0x00007f02640f9800 nid=0x3e36 runnable "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f026401d800 nid=0x3e34 runnable "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f026401f800 nid=0x3e35 runnable "VM Periodic Task Thread" os_prio=0 tid=0x00007f0264155000 nid=0x3e3d waiting on condition JNI global references: 309