WAITING at sun.misc.Unsafe.park(Native Method)

One of my applications hangs under some period of running under load, does anyone know what could cause such output in jstack:

"scheduler-5" prio=10 tid=0x00007f49481d0000 nid=0x2061 waiting on condition [0x00007f494e8d0000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000006ee117310> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

I am seeing this a lot in jstack output when it hangs.

I heavily using Spring @Async & maps, synchronized maps & ehcache.

What is interesting this only happens on one of app instances. Two others are running perfectly fine. What else I could investigate to get more details in such case?

I found this post https://stackoverflow.com/questions/23992787/parking-to-wait-for-0xd8cf0070-a-java-util-concurrent-locks-abstractqueueds but it is not very useful in my case.

unsafe.park is pretty much the same as thread.wait, except that it's using architecture specific code (thus the reason it's 'unsafe'). unsafe is not made available publicly, but is used within java internal libraries where architecture specific code would offer significant optimization benefits. It's used a lot for thread pooling.

So, to answer your question, all the thread is doing is waiting for something, it's not really using any CPU. Considering that your original stack trace shows that you're using a lock I would assume a deadlock is going on in your case.

Yes I know you have almost certainly already solved this issue by now. However, you're one of the top results if someone googles sun.misc.unsafe.park. I figure answering the question may help others trying to understand what this method that seems to be using all their CPU is.

From the stack trace it's clear that, the ThreadPoolExecutor > Worker thread started and it's waiting for the task to be available on the BlockingQueue(DelayedWorkQueue) to pick the task and execute.So this thread will be in WAIT status only as long as get a SIGNAL from the publisher thread.

I had a similar issue, and following previous answers (thanks!), I was able to search and find how to handle correctly the ThreadPoolExecutor terminaison.

In my case, that just fix my progressive increase of similar blocked threads:

• I've used ExecutorService::awaitTermination(x, TimeUnit) and ExecutorService::shutdownNow() (if necessary) in my finally clause.
• For information, I've used the following commands to detect thread count & list locked threads:

ps -u javaAppuser -L|wc -l
   
jcmd \`ps -C java -o pid=\` Thread.print >> threadPrintDayA.log
    
jcmd \`ps -C java -o pid=\` Thread.print >> threadPrintDayAPlusOne.log
    
cat threadPrint*.log |grep "pool-"|wc -l