Hanging java applications can be a system architects worst nightmare. There are so many pieces in the stack, finding where the true issue lies can be a daunting task. Fortunately, you can dump all threads in a JVM to see where it may be hanging.
DBA’s specifically should be able to read a thread dump, as the finger will often be pointed at the database as the source of the problem, when that may not be the case. If you can’t find any issues in the database that would indicate it could be the source of a hang, you should immediately ask for a thread dump in the application. A lot of times ( I have personally witnessed it ), the application support team will look at you with a blank stare and you won’t hear anymore about it (note this isn’t good for your customer if the problem isn’t fixed 🙁 )
To give you an idea as to what this may look like, below is an example of a class that fabricates a blocking thread within a multi-threaded java application. We then show how to prove this using a thread dump.
class syncCounter {
private int c = 0;
public synchronized void increment() {
try {
Thread.sleep(10000);
c++;
}
catch(Exception e) {}
}
}
class syncRun implements Runnable {
Thread t;
static syncCounter s;
public static void main(String args[]) throws Exception {
s = new syncCounter();
for (int i = 1; i <= 10; i++) {
syncRun tt = new syncRun("Thread " + i);
}
}
syncRun(String tname) {
t = new Thread(this,tname);
t.start();
}
public void run() {
s.increment();
System.out.println("Running thread " + t.getName());
}
}
When we run this application at the command line, the first thread locks the object monitor (the synchronized block of code above) for ten seconds before releasing it. The other nine threads will be blocked by this thread. The next thread to get the lock will block the next eight, and so on. The tenth thread will be blocked from running for at least 100 seconds as a result.
We compile the class and run it. After this, we find the java process by running the jps executable, and then run jstack against it and redirect the output to a file. Note you could also issue a kill -QUIT on *nix platforms to generate the stack trace.
linux5:root:orcl11g:/root>jps
28349 syncRun
28366 Jps
linux5:root:orcl11g:/root>jstack 28349 > stack.txt
When we review the stack.txt file, we see one thread something similar to the following...
"Thread 8" prio=6 tid=0x02aac000 nid=0x166c waiting on condition [0x0304f000..0x0304fb94]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at syncCounter.increment(syncCounter.java:6)
- locked <0x229bd478> (a syncCounter)
at syncRun.run(syncCounter.java:42)
at java.lang.Thread.run(Thread.java:619)
...and the nine other threads created by our sample program, each of which look like...
"Thread 7" prio=6 tid=0x02aaac00 nid=0x1330 waiting for monitor entry [0x02fff000..0x02fffd14]
java.lang.Thread.State: BLOCKED (on object monitor)
at syncCounter.increment(syncCounter.java:6)
- waiting to lock <0x229bd478> (a syncCounter)
at syncRun.run(syncCounter.java:42)
at java.lang.Thread.run(Thread.java:619)
If you notice, all nine threads blocked will show a "waiting to lock" for the same address as the thread that has the lock. If you can prove this, you just need to review the java code (and specific lines) noted in the thread dump and ascertain why it is blocking.