While troubleshooting a java thread that appeared to be hung, but not blocked, we used the following troubleshooting process.<\/p>\n
You can use what is below if you have a thread \u201cstuck\u201d waiting to receive data from another socket.<\/p>\n
We see the following thread, which we verified had an unchanging call stack over several thread dumps\u2026<\/p>\n
\r\n-bash-4.1$ jstack 24748 | awk '{if ($1 == \"\\\"Thread-3031\\\"\") {i = 1;print $0} else if (i == 1 && $0 != \"\") {print $0} else if (i == 1 && $0 == \"\") {exit}}'\r\n\"Thread-3031\" daemon prio=10 tid=0x00007fbeb8746800 nid=0x4cf3 runnable [0x00007fbe9aec8000]\r\n java.lang.Thread.State: RUNNABLE\r\n at java.net.SocketInputStream.socketRead0(Native Method)\r\n at java.net.SocketInputStream.read(SocketInputStream.java:129)\r\n at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)\r\n at java.io.BufferedInputStream.read(BufferedInputStream.java:237)\r\n - locked <0x0000000732c44848> (a java.io.BufferedInputStream)\r\n at java.io.DataInputStream.readByte(DataInputStream.java:248)\r\n at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:195)\r\n at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142)\r\n at atg.deployment.agent.RMIClientAgentTransport_Stub.sendCommand(Unknown Source)\r\n at atg.deployment.server.RMIServerAgentTransport.sendCommand(RMIServerAgentTransport.java:244)\r\n at atg.deployment.server.AgentRef.sendCommand(AgentRef.java:775)\r\n at atg.deployment.server.AgentRef.activateDeployment(AgentRef.java:1056)\r\n at atg.deployment.server.Deployment.activateDeployment(Deployment.java:5300)\r\n at atg.deployment.server.Deployment.run(Deployment.java:1962)\r\n at java.lang.Thread.run(Thread.java:662)\r\n<\/pre>\n\u2026so we convert the java thread id to an integer PID\u2026<\/p>\n
\r\n-bash-4.1$ printf \"%i\\n\" 0x4cf3\r\n19699\r\n<\/pre>\n\u2026then run strace to see what it is doing\u2026<\/p>\n
\r\n-bash-4.1$ strace -p 19699\r\nProcess 19699 attached - interrupt to quit\r\nrecvfrom(854, ^C \r\nProcess 19699 detached\r\n<\/pre>\n\u2026We see it waiting to receive data on file descriptor 854. As such, we then get the socket number for the file descriptor (visible in either the thread PID or the main java PID)\u2026<\/p>\n
\r\n-bash-4.1$ ls -lrt \/proc\/24748\/fd | grep 854\r\nlrwx------. 1 sa-jboss domain users 64 Jun 21 10:54 854 -> socket:[169393703]\r\n-bash-4.1$\r\n<\/pre>\n\u2026and finally, get the servers associated with this socket\u2026<\/p>\n
\r\n-bash-4.1$ netstat -e | grep 169393703\r\ntcp 0 0 server1.:39054 server2_with_problem.:50858 ESTABLISHED sa-jboss 169393703 24748\/java\r\n-bash-4.1$\r\n<\/pre>\nIn this case, we see server1 was hung waiting on data from server2.<\/p>\n
We found there were blocked threads on server2, and restarted it to ultimately resolve the issue.<\/p>\n","protected":false},"excerpt":{"rendered":"
While troubleshooting a java thread that appeared to be hung, but not blocked, we used the following troubleshooting process. You can use what is below if you have a thread \u201cstuck\u201d waiting to receive data from another socket. We see…<\/p>\n