A hung ThreadPoolExecutor in java can be manifested in various ways. At least one, that I wanted to post for my own notes, is below.
A thread dump may show something like what is below:
"pool-9-thread-5" prio=6 tid=0x01b84400 nid=0x16e0 waiting on condition [0x04fcf000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x236d77e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
"pool-9-thread-4" prio=6 tid=0x01b85c00 nid=0x1d98 waiting on condition [0x04f2f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x236d77e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
"pool-9-thread-3" prio=6 tid=0x01b85000 nid=0x58c waiting on condition [0x0501f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x236d77e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
"pool-9-thread-2" prio=6 tid=0x01b86400 nid=0x16c8 waiting on condition [0x050bf000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x236d77e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
"pool-9-thread-1" prio=6 tid=0x01b83000 nid=0x1ac0 waiting on condition [0x04d9f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x236d77e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
This was created by a simple java class shown below:
import java.io.*;
import java.util.*;
import java.util.concurrent.*;
class myPool implements Runnable {
Thread t;
public static void main(String args[]) {
for(int i=1; i <= Integer.parseInt(System.getProperty("pools")); i++){
ArrayBlockingQueue queue = new ArrayBlockingQueue(5);
ThreadPoolExecutor tpe = new ThreadPoolExecutor(5, 5, 5000, TimeUnit.SECONDS, queue);;
for (int j = 1; j <= Integer.parseInt(System.getProperty("threads")); j++){
tpe.execute(new myPool());
}
//uncomment to cleanly close the thread pool
//tpe.shutdown();
}
}
myPool() {
t = new Thread(this);
t.start();
}
public void run() {
try {
}
catch (Exception e) {
e.printStackTrace();
}
}
}
Notice how in our earlier thread dump each thread was "parking to wait for" an object with the same address. However, this was not a "locked" object. It is the queue passed to the ThreadPoolExecutor that it uses to pop and push threads as it executes them.
Since it is never shutdown, it will never be released, and in our case, will retain the five threads in the pool.
We found a similar issue, when we noticed a given JVM had 25,000 threads, each of which had a 256K stack size. If you do the math, we were using over 6GB of RAM on the OS (not in the heap), that was causing the server to start swapping pages to and from disk.
Simply issuing a shutdown() on the ThreadPoolExecutor resolved the issue.
2 comments for “One way to tell if a thread pool is hung”