There are a lot of GUI tools to process the call stacks for a “thread dump” taken on a running JVM. This is just a command line tool to do it, which is normally where I start troubleshooting when there is an issue.
In this case, these are simply “parent blockers”, for lack of a better word. This simply means those threads that are blocking another thread, when they themselves are not blocked. When a thread isn’t blocked, but is blocking another thread (or many threads), you need to look closely at the blockers call stack. This will enable you to determine why it has a lock on a synchronized block code, and what is preventing it from exiting the monitor. It could be a database lock, hung NFS filesystem, etc. The call stack will usually show you where the issue lies.
import sys
locked = dict()
stacks = dict()
addresses = dict()
done = dict()
def loadStackTraces():
thisStack = list()
next = True
for line in open(sys.argv[1],"r"):
if next == True:
thisThread = line.split()[0]
next = False
if len(line) == 1: #only has newline character, so line is functionally empty
stacks[thisThread] = thisStack
thisStack = list()
next = True
else:
thisStack.append(line)
if line.find("waiting to lock") > -1:
addresses[thisThread] = line.split()[4]
elif line.find("locked") > -1:
locked[thisThread] = line.split()[2]
loadStackTraces()
for i in addresses:
for j in locked:
if locked[j] == addresses[i]:
if j not in addresses:
if j not in done:
print '\n-----------------------------------------------------------------------------------------\n'
print j, " is a blocker, but is not blocked itself. Investigate the thread stack below:"
print '\n-----------------------------------------------------------------------------------------\n'
k = 0
for line in stacks[j]:
if k > 9:
break
print line.strip()
k = k + 1
done[j] = 1
The following example run shows the first ten lines of a problematic thread…
-bash-4.1$ python threadTree.py dump.txt
-----------------------------------------------------------------------------------------
"http-172.28.8.60-10543-5" is a blocker, but is not blocked itself. Investigate the thread stack below:
-----------------------------------------------------------------------------------------
"http-172.28.8.60-10543-5" daemon prio=10 tid=0x00007feff8005800 nid=0x5cad in Object.wait() [0x00007ff029ee6000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000534ae34d8> (a atg.service.lockmanager.ClientLockEntry)
at atg.service.lockmanager.ClientLockEntry.acquireWriteLock(ClientLockEntry.java:421)
- locked <0x0000000534ae34d8> (a atg.service.lockmanager.ClientLockEntry)
at atg.service.lockmanager.ClientLockManager.acquireWriteLock(ClientLockManager.java:1167)
at atg.service.lockmanager.ClientLockManager.acquireWriteLock(ClientLockManager.java:1263)
at atg.commerce.gifts.GiftlistManager.removeItemFromGiftlist(GiftlistManager.java:1042)
at com.express.order.purchase.ExpressCartFormHandler.postAddItemToOrder(ExpressCartFormHandler.java:3756)
-bash-4.1$ exit