We had a server that had been behaving oddly since at least April. I finally discovered why.<\/p>\n
jps returns nothing, which is the first symptom. When we strace, we see it can\u2019t write to \/tmp (our setting for java.io.tmpdir), so we look at permissions\u2026<\/p>\n
\r\n-bash-4.1$ strace -f jps 2>&1 | grep ACC\r\n[pid 16055] lstat(\"\/var\/lib\/samba\/winbindd_privileged\/pipe\", 0x7f3e9da9e170) = -1 EACCES (Permission denied)\r\n[pid 16055] mkdir(\"\/tmp\/hsperfdata_sa-jboss\", 0755) = -1 EACCES (Permission denied)\r\n-sh-4.1$ uname -n\r\nlck01.ecomm.local\r\n-sh-4.1$ ls -lart \/tmp | awk '$NF == \".\"'\r\ndrwxr-xr-x. 9 root root 4096 Aug 12 14:32 .\r\n<\/pre>\n\u2026the permissions on lck02 have the sticky bit set as well as world writable permissions\u2026<\/p>\n
\r\n-sh-4.1$ uname -n\r\nlck02.ecomm.local\r\n-sh-4.1$ ls -lart \/tmp | awk '$NF == \".\"'\r\ndrwxrwxrwt. 8 root root 4096 Aug 12 14:33 .\r\n<\/pre>\n\u2026so we mimic these on lck01\u2026<\/p>\n
\r\nroot@lck01 Tue Aug 12 14:33:30 EDT 2014\r\n[~] # chmod 777 \/tmp\r\n\r\nroot@lck01 Tue Aug 12 14:33:30 EDT 2014\r\n[~] # chmod o+t \/tmp\r\n\r\n-bash-4.1$ jps\r\n16129 Jps\r\n<\/pre>\nWe had stuck threads and everything else. The entire thing now works, no stuck threads, etc.<\/p>\n
\r\n-bash-4.1$ jps\r\n16370 Main\r\n16805 Jps\r\n-bash-4.1$ jstack 16370 | grep State: | sort | uniq -c\r\n 35 java.lang.Thread.State: RUNNABLE\r\n 14 java.lang.Thread.State: TIMED_WAITING (on object monitor)\r\n 5 java.lang.Thread.State: TIMED_WAITING (parking)\r\n 8 java.lang.Thread.State: TIMED_WAITING (sleeping)\r\n 46 java.lang.Thread.State: WAITING (on object monitor)\r\n 83 java.lang.Thread.State: WAITING (parking)\r\n-bash-4.1$\r\n<\/pre>\nThe moral of the story is, check the permissions on \/tmp if you have unexplained behavior from a java application.<\/p>\n","protected":false},"excerpt":{"rendered":"
We had a server that had been behaving oddly since at least April. I finally discovered why. jps returns nothing, which is the first symptom. When we strace, we see it can\u2019t write to \/tmp (our setting for java.io.tmpdir), so…<\/p>\n