Why can’t I truncate my active JBOSS server.log file?

If you find yourself in a position where it is 3AM and the filesystem on which your server.log file lives is full, you may issue some kind off redirect or “cat /dev/null > server.log”, or even “truncate –size=0 server.log”. After congratulating yourself on knowing you can’t simply delete a file that is attached to an open process, to your horror you notice the file is still growing.

The issue is that the server.log file has been opened without the O_APPEND flag. This flag tells the process that whenever a write occurs, it checks with the OS to find the actual current end of the file. If you don’t pass this flag, the process keeps track of where it was at the last write, and will simply pick up there and issue the next write. As such, if the file was 4236754986 bytes at the last write, you can truncate the file, and the process will simply fseek() to 4236754987 bytes and start writing.

To fix, you can use the JMX to stop logging and roll the file.

http://jboss.example.com:8080/jmx-console/HtmlAdaptor?action=inspectMBean&name=jboss.system%3Atype%3DLog4jService%2Cservice%3DLogging

You can also configure maxLogFileSize in the logging config.

import java.io.*;

public class c {
  public static void main(String args[]) throws Exception {
    StringBuffer sb = new StringBuffer();
    for (int i = 1; i <= 100000; i++)
      sb.append("**********");
    String s = sb.toString();
    System.out.println("starting write");
    FileWriter f = new FileWriter("test.txt");
    for (int j = 1; j <= 5000; j++) {
      f.write(s);
      f.flush();
    }
  }
}

[adm-showard@cmhlcarchapp01 ~]$ java c &
[adm-showard@cmhlcarchapp01 ~]$ for i in {1..5}; do > test.txt; done
[adm-showard@cmhlcarchapp01 ~]$ ls -lrt test.txt
-rw-r--r-- 1 adm-showard domain users 5000000000 Oct 28 10:03 test.txt
[adm-showard@cmhlcarchapp01 ~]$ du test.txt
1606280 test.txt

The filesystem will still show the space as in use.

You can see which flags were used in the open call with what is below...

[/app/express/jboss/server/loadtestecm01/log] # ps -ef | grep java | grep -v grep | awk '{print $2}'
2138
15066

root@atllpecomecm04 Sat Aug 11 11:14:42 EDT 2018
[/app/express/jboss/server/loadtestecm01/log] # ls -lrt /proc/2138/fd | grep server.log
l-wx------. 1 sa-jboss domain users 64 Aug  9 17:06 72 -> /app/express/jboss/server/ecm01/log/server.log

root@atllpecomecm04 Sat Aug 11 11:14:42 EDT 2018
[/app/express/jboss/server/loadtestecm01/log] # cat /proc/2138/fdinfo/72 | head -10
pos:    692940719
flags:  0100001

root@atllpecomecm04 Sat Aug 11 11:14:42 EDT 2018
[/app/express/jboss/server/loadtestecm01/log] # 

The flags line is an octal representation of O_WRONLY|O_LARGEFILE, so this file was not opened with the O_APPEND flag set.