\u2026so we get the PID for this thread\u2026<\/p>\n
\r\n-bash-4.1$ printf \"%i\\n\" \"0x8f1\"\r\n2289\r\n<\/pre>\n\u2026then trace the PID and see we are waiting in perpetuity on a recvfrom() call on a network socket\u2026<\/p>\n
\r\n-bash-4.1$ strace -p 2289\r\nProcess 2289 attached - interrupt to quit\r\nrecvfrom(1040, ^C\r\nProcess 2289 detached\r\n<\/pre>\n \u2026so we get the file descriptor on which it is waiting\u2026<\/p>\n
\r\n-bash-4.1$ ls -lrt \/proc\/2289\/fd | grep 1040\r\nlrwx------. 1 sa-jboss domain users 64 Jun 25 11:32 1040 -> socket:[849616076]\r\n<\/pre>\n\u2026and we then see it is a socket, so we need to see where is it connecting\u2026<\/p>\n
\r\n-bash-4.1$ netstat -aenp | grep 849616076\r\n(Not all processes could be identified, non-owned process info\r\nwill not be shown, you would have to be root to see it all.)\r\ntcp 0 0 1.2.2.6:57779 10.0.8.2:5555 ESTABLISHED 11006 849616076 32384\/java\r\n<\/pre>\n\u2026and we see the IP is our order management vendor.<\/p>\n
We immediately assumed the sender was hung. However, when we dug deeper using tcpdump and wireshark, we saw we had enabled path MTU discovery in the Linux host OS. <\/p>\n
tcpdump capture…<\/p>\n
\r\n[~] # tcpdump host 54.208.67.100 -tttt -s 65535 -w hung.pcap\r\ntcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes\r\n^C1458 packets captured\r\n1470 packets received by filter\r\n0 packets dropped by kernel\r\n<\/pre>\nWireshark screen of the above pcap file…<\/p>\n
<\/p>\n
Enabling path MTU with the default value of 0 requires an ICMP reply to get the maximum MTU possible. It wasn’t getting this back, so our conjecture is that this was the issue.<\/p>\n
As soon as we set net.ipv4.ip_no_pmtu_disc = 1, disabling path MTU, our problems went away.<\/p>\n","protected":false},"excerpt":{"rendered":"
We received an email asking that we restart a given server as the ATG lock manager had a required lock and a scheduled job could not move forward. I don’t like to do that without some triage. We were stuck…<\/p>\n