Logrotate

Logrotate is a Unix utility to manage log files. I wish I had known how to use it much earlier as its very simple to use and makes adding log rotation easy for any project.

The problem it solves is when you log to a file and just constantly append to it. Eventually the log is going to become huge and it's going to be a pain to use to investigate problems. Logrotate will make a copy of the log file and start a new one based on a configuration you write.

Log rotation is done through 2 Linux utilities, the first is the crontab that manages running logrotate and then logrotate itself is used to manage renaming logs and copying them.

The first thing that happens is that the cron for logrotate needs to be set up. Logrotate by default will run daily.

/etc/crontab

25 6    * * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )

This line in the system crontab is saying to check if anacron is available. If it is, then crons will be run using anacron. I've never heard of anacron before and my system does have it but not at /usr/sbin/anacron. Will need to investigate.

The second part of this line is the important part. It is saying to run all the scripts found in /etc/cron.daily at 6:25am every day.

Logrotate runs as a system cron that you can find in /etc/cron.daily/logrotate.

/etc/cron.daily/logrotate

/usr/sbin/logrotate /etc/logrotate.conf

The key part of this script is that it runs the logrotate command using the logrotate.conf file.

/etc/logrotate.conf

include /etc/logrotate.d

The key part of this file is that we include all the scripts under logrotate.d. This means that if we want to write a new log rotation script, we can simply place it it under that folder and our log rotation will run once a day! Now if want our logs to rotate hourly, we would need to copy our /etc/cron.daily/logrotate to cron.hourly.

Now we can look at an example of a log rotation script!

/etc/logrotate.d/sample-log

/path/to/log/log.txt {
    nocompress
    copytruncate
    dateext
    rotate 15
    size 250K
}

This is simple script that works well for me. The first thing I do is get rid of compression, usually if I'm reading the logs, I want to just use vim and space isn't too much of a worry for me.

The next line is copytruncate which means that the log file will be copied first, then the original file will be truncated. This means that if the log is very active, there will be lost log lines. Luckily I don't have to care about this so I can do this.

The gain with using copytruncate is that the running application doesn't need to worry about log files disappearing or having to reboot the application to re-create log files.

The next is dateext which means to save the rotated log files with today's date. This way I can quickly tell what date the log files are from instead of just seeing .0, .1 and so on.

The fifth line says to keep only 15 logs. Once 15 is hit, it will start removing the oldest log file.

Lastly the size option lets you set the size at which to rotate. Here this means that at 250k kilobytes, rotate the file by copying it and truncating the original to 0. Name the rotated file with the date as the extension and don't compress it. Keep 15 of these back up logs for posterity.

There we have it! the entire flow of how logrotate works from a Linux point of view and an example of a simple rotation script that works. Now our application doesn't need to worry about managing the log file and we don't need to figure out how to read and investigate 10 GB log files!