A hung ThreadPoolExecutor in java can be manifested in various ways. At least one, that I wanted to post for my own notes, is below.
A thread dump may show something like what is below:
"pool-9-thread-5" prio=6 tid=0x01b84400 nid=0x16e0 waiting on condition [0x04fcf000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x236d77e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
"pool-9-thread-4" prio=6 tid=0x01b85c00 nid=0x1d98 waiting on condition [0x04f2f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x236d77e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
"pool-9-thread-3" prio=6 tid=0x01b85000 nid=0x58c waiting on condition [0x0501f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x236d77e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
"pool-9-thread-2" prio=6 tid=0x01b86400 nid=0x16c8 waiting on condition [0x050bf000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x236d77e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
"pool-9-thread-1" prio=6 tid=0x01b83000 nid=0x1ac0 waiting on condition [0x04d9f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x236d77e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
This was created by a simple java class shown below:
import java.io.*;
import java.util.*;
import java.util.concurrent.*;
class myPool implements Runnable {
Thread t;
public static void main(String args[]) {
for(int i=1; i <= Integer.parseInt(System.getProperty("pools")); i++){
ArrayBlockingQueue queue = new ArrayBlockingQueue(5);
ThreadPoolExecutor tpe = new ThreadPoolExecutor(5, 5, 5000, TimeUnit.SECONDS, queue);;
for (int j = 1; j <= Integer.parseInt(System.getProperty("threads")); j++){
tpe.execute(new myPool());
}
//uncomment to cleanly close the thread pool
//tpe.shutdown();
}
}
myPool() {
t = new Thread(this);
t.start();
}
public void run() {
try {
}
catch (Exception e) {
e.printStackTrace();
}
}
}
Notice how in our earlier thread dump each thread was "parking to wait for" an object with the same address. However, this was not a "locked" object. It is the queue passed to the ThreadPoolExecutor that it uses to pop and push threads as it executes them.
Since it is never shutdown, it will never be released, and in our case, will retain the five threads in the pool.
We found a similar issue, when we noticed a given JVM had 25,000 threads, each of which had a 256K stack size. If you do the math, we were using over 6GB of RAM on the OS (not in the heap), that was causing the server to start swapping pages to and from disk.
Simply issuing a shutdown() on the ThreadPoolExecutor resolved the issue.
what did you to fix the issue?
Hi Safdar,
Like anything else, it depends. You need to determine if the threads are blocked by something else, or just slow. A thread dump taken with…
jstack YOUR_JVM_PID > debug.out
…will normally show exactly where the problem is.
There is no safe way to *kill* a thread in java (see this for more detail). If you can’t figure out if one thread is blocking several others due to a database lock, or hung network socket, it may require a restart. Just ensure you get at least three or four thread dumps, if possible. If it is hung enough, you may not even be able to get that.
The big thing is to fix the code if you have access, or go to the vendor armed with thread dumps if it is truly a code issue. I have also decompiled vendor code to see if it really is a code issue.