The easiest way to embed static files into a binary file in your Golang app (no external dependencies)

Introduction

Hello, everyone! 👋 I have a very interesting find for you. But first, I'm going to tell you a little background story, so...

Earlier, I tried go.rice and go-bindata, then I used packr and switched to pkger from the same author (Hi, Mark Bates, if you're suddenly reading this). Yep, I was satisfied with this packages and quite happy!

But when I saw this method somewhere on the Internet, I immediately herded it into all of my Golang apps (who needed it, of course).

...and what's problem with ready-made packages?

First of all, it's external packages with their dependencies. It's good when you need to quickly make a working prototype OR don't really care about the size of your code base.

Documentation may not be complete or cover actually what you need at this moment. I have been frustrated to have to wait a long time (sometimes, up to several weeks) to get an answer OR a bug fix.

I have nothing against the authors of these great Go packages! I personally support many of Open Source repositories on GitHub and understand they very well! 😉

Disclaimer

This article was written before Go 1.16 released, which added awesome features for adding files to the final binary.

Why I need this?

Good question, by the way, because many programmers do not always understand advantages of this approach to packaging a product (the final program, which you sell to your customer).

☝️ Especially, who came from scripting languages like JavaScript, Python, PHP or else.

I highlight the following:

• It's safe. Files, which added to a binary, are impossible (or extremely difficult) to extract back.

• It's convenient. There's no need to always remember to include a folder with static files, when you give production-ready product to your customer.

OK! So what am I suggesting? 🤔

My suggest will be very simple and elegant: why not write our own packer for static files using only built-in Go tools? Yes, gophers have already figured out what I'm talking about... Go generate function!

⭐️ Please, read first this awesome blog post by Rob Pike (22 Dec 2014) for more understand how it works! Unfortunately, I can't explain this principle better and shorter than Rob did...

No time for further reading? 🚀

I created repository with full code example on my GitHub especially for you:

koddr / example-embed-static-files-go

The easiest way to embed static files into a binary file in your Golang app (no external dependencies).

Just git clone and read instructions from README.

Project structure

Let's go mod init ... and create some files 👌

$ tree .
.
├── Makefile
├── go.mod
├── cmd
│   └── app
│       └── main.go          <-- main file of your Go app
├── internal
│   └── box                  <-- template and functions for create blob
│       ├── box.go
│       └── generator.go
└── static                   <-- folder with static files
    └── index.html

Add most used commands to Makefile:

.PHONY: generate

# ⚠️
generate:
    @go generate ./...
    @echo "[OK] Files added to embed box!"

security:
    @gosec ./...
    @echo "[OK] Go security check was completed!"

build: generate security
    @go build -o ./build/server ./cmd/app/*.go
    @echo "[OK] App binary was created!"

run:
    @./build/server

⚠️ I strongly recommend to use Go security checker for your code! Maybe I'll write a separate article about this great package later.

Create methods for use on code (box.go)

//go:generate go run generator.go

package box

type embedBox struct {
    storage map[string][]byte
}

// Create new box for embed files
func newEmbedBox() *embedBox {
    return &embedBox{storage: make(map[string][]byte)}
}

// Add a file to box
func (e *embedBox) Add(file string, content []byte) {
    e.storage[file] = content
}

// Get file's content
func (e *embedBox) Get(file string) []byte {
    if f, ok := e.storage[file]; ok {
        return f
    }
    return nil
}

// Embed box expose
var box = newEmbedBox()

// Add a file content to box
func Add(file string, content []byte) {
    box.Add(file, content)
}

// Get a file from box
func Get(file string) []byte {
    return box.Get(file)
}

All magic included into first line: //go:generate .... This call Go compiler to use methods from this file for code generate by another.

Create template and functions for blob file (generator.go)

//+build ignore

package main

import (
    "bytes"
    "fmt"
    "go/format"
    "io/ioutil"
    "log"
    "os"
    "path/filepath"
    "strings"
    "text/template"
)

const (
    blobFileName string = "blob.go"
    embedFolder  string = "../../static"
)

// Define vars for build template
var conv = map[string]interface{}{"conv": fmtByteSlice}
var tmpl = template.Must(template.New("").Funcs(conv).Parse(`package box

// Code generated by go generate; DO NOT EDIT.

func init() {{{ range $name, $file := . }}
        box.Add("{{ $name }}", []byte{ {{ conv $file }} }){{ end }}
}`),
)

func fmtByteSlice(s []byte) string {
    builder := strings.Builder{}

    for _, v := range s {
        builder.WriteString(fmt.Sprintf("%d,", int(v)))
    }

    return builder.String()
}

func main() {
    // Checking directory with files
    if _, err := os.Stat(embedFolder); os.IsNotExist(err) {
        log.Fatal("Configs directory does not exists!")
    }

    // Create map for filenames
    configs := make(map[string][]byte)

    // Walking through embed directory
    err := filepath.Walk(embedFolder, func(path string, info os.FileInfo, err error) error {
        relativePath := filepath.ToSlash(strings.TrimPrefix(path, embedFolder))

        if info.IsDir() {
            // Skip directories
            log.Println(path, "is a directory, skipping...")
            return nil
        } else {
            // If element is a simple file, embed
            log.Println(path, "is a file, packing in...")

            b, err := ioutil.ReadFile(path)
            if err != nil {
                // If file not reading
                log.Printf("Error reading %s: %s", path, err)
                return err
            }

            // Add file name to map
            configs[relativePath] = b
        }

        return nil
    })
    if err != nil {
        log.Fatal("Error walking through embed directory:", err)
    }

    // Create blob file
    f, err := os.Create(blobFileName)
    if err != nil {
        log.Fatal("Error creating blob file:", err)
    }
    defer f.Close()

    // Create buffer
    builder := &bytes.Buffer{}

    // Execute template
    if err = tmpl.Execute(builder, configs); err != nil {
        log.Fatal("Error executing template", err)
    }

    // Formatting generated code
    data, err := format.Source(builder.Bytes())
    if err != nil {
        log.Fatal("Error formatting generated code", err)
    }

    // Writing blob file
    if err = ioutil.WriteFile(blobFileName, data, os.ModePerm); err != nil {
        log.Fatal("Error writing blob file", err)
    }
}

At this point, we set //+build ignore on first line, because this code needed only for code generation process and don't be in main build.

We're almost done! 👍

Let's go through the block file generation process itself in more detail.

So, running command:

$ make generate

...it created for us blob.go file (on current project folder) with this code:

package box

// Code generated by go generate; DO NOT EDIT.

func init() {
    box.Add("/index.html", []byte{35, 32, ...}
    // ...
}

Basically, we converted the files in the ./static folder to a slice of bytes and now we can easily add them (in this clean form) to our binary executable! 🎉

Time to use it all in your code

package main

import (
    "log"
    "net/http"
    "text/template"

    "github.com/koddr/example-embed-static-files-go/internal/box"
)

type PageData struct {
    Title       string
    Heading     string
    Description string
}

func main() {
    // Define embed file (for a short)
    index := string(box.Get("/index.html"))

    // Template
    tmpl := template.Must(template.New("").Parse(index))

    // Handle function
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        data := PageData{
            Title:       "The easiest way to embed static files into a binary file",
            Heading:     "This is easiest way",
            Description: "My life credo is 'If you can not use N, do not use N'.",
        }

        // Execute template with data
        if err := tmpl.Execute(w, data); err != nil {
            log.Fatal(err)
        }
    })

    // Start server
    if err := http.ListenAndServe(":8080", nil); err != nil {
        log.Fatal(err)
    }
}

☝️ Always use / for looking up! For example: /images/logo.jpg is actually ./static/images/logo.jpg.

Exercises 🤓

• Improve box.Get() function for getting all files from embed folder at once call (for example, by using wildcard). Like box.Get("/images/*") or similar.
• Exclude path to directory of static files to flags and when call it on go:generate.

Final words

I understand, that it would be much better to take a ready-made library and use its API... but then you would never get to know this wonderful tool and write your first code generator!

Invest in yourself and your horizons! 🎁

Photo by

[Title] Vitor Santos https://unsplash.com/photos/GOQ32dlahDk
[1] Brunno Tozzo https://unsplash.com/photos/t32lrFimPlU
[2] Christin Hume https://unsplash.com/photos/Hcfwew744z4

P.S.

If you want more articles (like this) on this blog, then post a comment below and subscribe to me. Thanks! 😻

❗️ You can support me on Boosty, both on a permanent and on a one-time basis. All proceeds from this way will go to support my OSS projects and will energize me to create new products and articles for the community.

And of course, you can help me make developers' lives even better! Just connect to one of my projects as a contributor. It's easy!

My main projects that need your help (and stars) 👇

• 🔥 gowebly: A next-generation CLI tool that makes it easy to create amazing web applications with Go on the backend, using htmx, hyperscript or Alpine.js and the most popular CSS frameworks on the frontend.
• ✨ create-go-app: Create a new production-ready project with Go backend, frontend and deploy automation by running one CLI command.

Other my small projects: yatr, gosl, json2csv, csv2api.

Comments

[Fernando Guisso]

Hi, great article, I'll try to implement in my app.
2 questions:

• Do you have any best form to serve all static file automatic? because a have this one

static/css/*.css
static/image/*.png
static/js/*js

too many files to create one endpoint for each one and point the blob.

• If I have a lot of files in /static, all files is load when my app start? do you have problems with memory?

[Vic Shóstak]

Hello! 👋 Thanks for reply.

1. If you've many files to embed, you can create little helper func for its append to binary. For example, with filepath.Walk.

2. I'm not sure, but on my some huge React.js app it works perfectly, as usual. But, some benchmarks it's better... 🤔

[Fernando Guisso]

my solution:

for filename, content := range static.Map() {                               
         r.HandleFunc(fmt.Sprintf("/static%v", filename),                        
             func(w http.ResponseWriter, r *http.Request) {                      
                 http.ServeContent(w, r, "test.txt", time.Now(),                 
                 bytes.NewReader(content))                                                                  
         })                                                                                                 
}

static is my box pkg and the .Map just return the storages map

The problem now is because I use the html/template to generate some pages, and I can't use the funcs of this pkg :(

[Jose Diaz-Gonzalez]

Do you know of a decent way to use this pattern but also add support for either partials or layouts? I think thats maybe the only thing missing from this, and I don't think I'm quite at the level of coming up with my own solution :D

[Vic Shóstak]

Hi! Please write in more detail what is unclear and how you would like to improve or change something in the article. I will try to improve the article :)

[Kyli]

Great job. But placing the "//go:generate go run generator.go" in the generator.go a better choice?

[Vic Shóstak]

Thanks. Why not to do this? Please write in more detail what is unclear ;)