There are a lot of GUI tools to process the call stacks for a “thread dump” taken on a running JVM. This is just a command line tool to do it, which is normally where I start troubleshooting when there is an issue.<\/p>\n
In this case, these are simply “parent blockers”, for lack of a better word. This simply means those threads that are blocking another thread, when they themselves are not blocked. When a thread isn’t blocked, but is blocking another thread (or many threads), you need to look closely at the blockers call stack. This will enable you to determine why it has a lock on a synchronized block code, and what is preventing it from exiting the monitor. It could be a database lock, hung NFS filesystem, etc. The call stack will usually show you where the issue lies.<\/p>\n
\r\nimport sys\r\n\r\nlocked = dict()\r\nstacks = dict()\r\naddresses = dict()\r\ndone = dict()\r\n\r\ndef loadStackTraces():\r\n thisStack = list()\r\n next = True\r\n for line in open(sys.argv[1],\"r\"):\r\n if next == True:\r\n thisThread = line.split()[0]\r\n next = False\r\n if len(line) == 1: #only has newline character, so line is functionally empty\r\n stacks[thisThread] = thisStack\r\n thisStack = list()\r\n next = True\r\n else:\r\n thisStack.append(line)\r\n if line.find(\"waiting to lock\") > -1:\r\n addresses[thisThread] = line.split()[4]\r\n elif line.find(\"locked\") > -1:\r\n locked[thisThread] = line.split()[2]\r\n\r\nloadStackTraces()\r\nfor i in addresses:\r\n for j in locked:\r\n if locked[j] == addresses[i]:\r\n if j not in addresses:\r\n if j not in done:\r\n print '\\n-----------------------------------------------------------------------------------------\\n'\r\n print j, \" is a blocker, but is not blocked itself. Investigate the thread stack below:\"\r\n print '\\n-----------------------------------------------------------------------------------------\\n'\r\n k = 0\r\n for line in stacks[j]:\r\n if k > 9:\r\n break\r\n print line.strip()\r\n k = k + 1\r\n done[j] = 1\r\n<\/pre>\nThe following example run shows the first ten lines of a problematic thread…<\/p>\n
\r\n-bash-4.1$ python threadTree.py dump.txt\r\n\r\n-----------------------------------------------------------------------------------------\r\n\r\n\"http-172.28.8.60-10543-5\" is a blocker, but is not blocked itself. Investigate the thread stack below:\r\n\r\n-----------------------------------------------------------------------------------------\r\n\r\n\"http-172.28.8.60-10543-5\" daemon prio=10 tid=0x00007feff8005800 nid=0x5cad in Object.wait() [0x00007ff029ee6000]\r\njava.lang.Thread.State: TIMED_WAITING (on object monitor)\r\nat java.lang.Object.wait(Native Method)\r\n- waiting on <0x0000000534ae34d8> (a atg.service.lockmanager.ClientLockEntry)\r\nat atg.service.lockmanager.ClientLockEntry.acquireWriteLock(ClientLockEntry.java:421)\r\n- locked <0x0000000534ae34d8> (a atg.service.lockmanager.ClientLockEntry)\r\nat atg.service.lockmanager.ClientLockManager.acquireWriteLock(ClientLockManager.java:1167)\r\nat atg.service.lockmanager.ClientLockManager.acquireWriteLock(ClientLockManager.java:1263)\r\nat atg.commerce.gifts.GiftlistManager.removeItemFromGiftlist(GiftlistManager.java:1042)\r\nat com.express.order.purchase.ExpressCartFormHandler.postAddItemToOrder(ExpressCartFormHandler.java:3756)\r\n-bash-4.1$ exit\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"There are a lot of GUI tools to process the call stacks for a “thread dump” taken on a running JVM. This is just a command line tool to do it, which is normally where I start troubleshooting when there…<\/p>\n