Time based log file rotation with zap

zap is one of the nicely maintained, well-written and nicely performing open source logging libraries in Go. Unfortunately, not all apps in our organization are 12-factor apps and thus we still log to physical files. So, I needed a way to rotate log files while using zap. I come from Java background, and log4j has a very vast array of log rotation options. But I found it lacking in zap, clearly because in the world of containers, logging to stdout (treating logs as event streams - 12-factor logs) is very common.

To be specific, I was looking to do time based rotation (one log file per hour, helps in debugging) and not size based. On zap's FAQ, they've shown integration with lumberjack, which only supports size based rotation.

I found another library file-rotatelogs which does support time based rotation. So I used it along with zap.

The good thing about zap is that it takes any io.Writer interface as a WriteSyncer and file-rotatelogs returns *RotateLogs which implements the io.Writer interface.

Here the rotatelogs.WithRotationTime(time.Hour)) indicates hourly rotation. rotatelogs.WithMaxAge(60*24*time.Hour) indicates the files are cleaned up after 60 days.

package main

import (
    "encoding/json"
    "time"

    rotatelogs "github.com/lestrrat-go/file-rotatelogs"
    "go.uber.org/zap"
    "go.uber.org/zap/zapcore"
)

func main() {
    // initialize the rotator
    logFile := "/var/log/app-%Y-%m-%d-%H.log"
    rotator, err := rotatelogs.New(
        logFile,
        rotatelogs.WithMaxAge(60*24*time.Hour),
        rotatelogs.WithRotationTime(time.Hour))
    if err != nil {
        panic(err)
    }

    // initialize the JSON encoding config
    encoderConfig := map[string]string{
        "levelEncoder": "capital",
        "timeKey":      "date",
        "timeEncoder":  "iso8601",
    }
    data, _ := json.Marshal(encoderConfig)
    var encCfg zapcore.EncoderConfig
    if err := json.Unmarshal(data, &encCfg); err != nil {
        panic(err)
    }

    // add the encoder config and rotator to create a new zap logger
    w := zapcore.AddSync(rotator)
    core := zapcore.NewCore(
        zapcore.NewJSONEncoder(encCfg),
        w,
        zap.InfoLevel)
    logger := zap.New(core)

    logger.Info("Now logging in a rotated file")
}

This will create the file /var/log/app-2020-05-18-15.log, which is exactly how we wanted it. It will automatically start logging to the next file during the start of the next hour.

Comments

[Simme]

Hi, and welcome to dev.to! 👋🏼

To keep the user experience pleasant and allowing users to browse interesting content without having to leave the site, we encourage authors to post their content here rather than to link to external sources.

I understand, though, that you might want the posts to originate from your own blog. I feel the same way, which is why I use the canonical URL property on my posts and post them in their entirety. You can actually even set up your dev.to account to sync posts from an external RSS feed.

See dev.to/settings/publishing-from-rss for more info. 👍🏼

Best regards,
Simme

[Dhwaneet Bhatt]

Thanks for the detailed explanation Simon.

[Hariprasad]

It's a good one but i won't prefer that. If you move your logs to any database for analytical purpose is much appreciated than this.

[Dhwaneet Bhatt]

We are preparing our log stack, but it will take time. We are still early stages of a startup and for now, we use files for debugging and finding out errors.