Tying a linux socket file descriptor to a port and IP

While troubleshooting a java thread that appeared to be hung, but not blocked, we used the following troubleshooting process.

You can use what is below if you have a thread “stuck” waiting to receive data from another socket.

We see the following thread, which we verified had an unchanging call stack over several thread dumps…

-bash-4.1$ jstack 24748 | awk '{if ($1 == "\"Thread-3031\"") {i = 1;print $0} else if (i == 1 && $0 != "") {print $0} else if (i == 1 && $0 == "") {exit}}'
"Thread-3031" daemon prio=10 tid=0x00007fbeb8746800 nid=0x4cf3 runnable [0x00007fbe9aec8000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
        - locked <0x0000000732c44848> (a java.io.BufferedInputStream)
        at java.io.DataInputStream.readByte(DataInputStream.java:248)
        at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:195)
        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142)
        at atg.deployment.agent.RMIClientAgentTransport_Stub.sendCommand(Unknown Source)
        at atg.deployment.server.RMIServerAgentTransport.sendCommand(RMIServerAgentTransport.java:244)
        at atg.deployment.server.AgentRef.sendCommand(AgentRef.java:775)
        at atg.deployment.server.AgentRef.activateDeployment(AgentRef.java:1056)
        at atg.deployment.server.Deployment.activateDeployment(Deployment.java:5300)
        at atg.deployment.server.Deployment.run(Deployment.java:1962)
        at java.lang.Thread.run(Thread.java:662)

…so we convert the java thread id to an integer PID…

-bash-4.1$ printf "%i\n" 0x4cf3
19699

…then run strace to see what it is doing…

-bash-4.1$ strace -p 19699
Process 19699 attached - interrupt to quit
recvfrom(854, ^C 
Process 19699 detached

…We see it waiting to receive data on file descriptor 854. As such, we then get the socket number for the file descriptor (visible in either the thread PID or the main java PID)…

-bash-4.1$ ls -lrt /proc/24748/fd | grep 854
lrwx------. 1 sa-jboss domain users 64 Jun 21 10:54 854 -> socket:[169393703]
-bash-4.1$

…and finally, get the servers associated with this socket…

-bash-4.1$ netstat -e | grep 169393703
tcp        0      0 server1.:39054 server2_with_problem.:50858 ESTABLISHED sa-jboss   169393703 24748/java
-bash-4.1$

In this case, we see server1 was hung waiting on data from server2.

We found there were blocked threads on server2, and restarted it to ultimately resolve the issue.