Procedural Pixel Art Tilemaps

Hi. I've been developing Medieval Life for the past 2 years. It's still in it's early stages because I'm taking this project as a hobby project.

The game is a procedurally generated game. It means all the terrain is generated randomly when the world is first created.
The game will have a few different textures for the terrain (for example, grass, sand, mud, rocks, wood path, and so on.) and the player will be able to edit the type of texture for each terrain cell.

The idea is that no matter which combination of tiles the player uses, the game will be able to create smooth pixel art transitions.

Each tile, has 8 neighbour tiles. As you can see, if you want to create every posible combination by hand the number of tilemaps required would be exponentially huge.

That's why I created a system to create procedural pixel art tilemaps on the fly.

The source tilemaps

The first thing to do is to think a way to reduce the number of manually required tile pieces to draw.

I stumbled across this amazing tweet by Oskar Stålberg:

I recommend you to click it and give it a look. Basically, what it shows is a change of mind in the way we design tilemaps: Instead of having a tile with 8 neighbour cells, we shift our graphical tilemap half a unit and now we only need to consider 4 neightbour cells per tile.

So, I designed all my tilesets using this method. Below there are some examples:

If you look closely, you will find a few things:

First, each tile has only 4 neighbors: The North-west, north-east, south-west and south-east.

Secondly, the position of each tile in the tilemap follows a binary order. You can encode each neighbor (NW/NE/SW/SE) as a 4 bit binary digit. For example, the first tile will be 0000 (all "off"). The second tile will be 0001 (only SE). The third tile will be 0010 (only SW), the fourth 0011 (SE and SW), then 0100, 0101, and so on, until the last tile that is full 1111, that means every neighbour is "on". This particular order will come handy later.

Here is a visual example:

The third thing to note is every tile has a transparent pink irregular outline. This is what I call the "shadow zone" and It is used to tell the game which pixels would normally be in shadow when generating the procedural tilemap. I will talk more about later.

The TileRequest and TextureManager

Every time the terrain mesh builder needs a tile texture to build the mesh of the terrain, it creates a new TileRequest
that is sent to the TextureManager. The tile request is a simple struct that has 4 fields, one for each neighbour:

public readonly struct TileRequest
{
    public Cover CoverNorthWest;
    public Cover CoverNorthEast;
    public Cover CoverSouthWest;
    public Cover CoverSouthEast;
}

In my current implementation I use 9 fields instead of 4, (8 fields for each of the neighbours and one for the central tile), but I will change it to 4 later because it's better and it outputs less tiles combinations.

Cover is a class that represents the kind of terrain, it can be "Wood", "Sand", "Grass", etc. But you can use a struct or any way or representing the different kinds of terrain.

For example think the following complex case. The marked area is the "visual tile" (Remember that Visual tiles have an offset of half unit respect to the logical tiles used at game play)

In this example, we have the following:

• CoverNorthWest: Stone
• CoverNorthEast: Wood
• CoverSouthWest: Snow
• CoverSouthEast: Grass

After the TileRequest is created, the TextureManager searches in a Dictionary<TileRequest, PackedTexture> if there is already an existing PackedTexture (visual tile image inside a texture atlas) for the given TileRequest. This check is super fast, because it's only checking for a value inside a dictionary.

If there is no PackedTexture available for that request, then it means it's a new tile that has never been generated before and the TextureManager needs to generate that tile image.

To generate the image the steps are the following:

1. Request a new space for the tile into the Texture Atlas. If there are no enough space in the atlas, the system needs to create a new Atlas image. A new PackedTexture is created that contains the information about the texture of the atlas and the UVs of the rectangle.
2. Then transform the TileRequest to a TileDrawOperation (I will talk about this later) and draw the image in the atlas.
3. Mark the atlas as dirty so the atlas texture it's uploaded to the GPU at the beginning of the next frame.
4. Add the TileRequest and PackedTexture to the Dictionary, so it when the same TileRequest is requested a second time, the texture is already there.
5. Finally, return the PackedTexture.

The final texture atlas looks something like this:

I'm currently using 8 neightbours instead of the 4 described in the tweet by Oskar Stålberg. If you use 4 the number of tile combinations will be much lower. I also reuse the same texture atlas to draw the walls, roofs and textures of trees, bushes, water and furniture. That's why you see other textures in the atlas.

The TileDrawOperation

The last part is to draw the procedural tile itself. I draw the Tile on the CPU because the images are pixel art and super small, and I need a finer grained control over the final image. I don't think it's worth to do it on the GPU. I also process every pixel in parallel with System.Threading.Tasks.Parallel.For. So It's extremely fast. I think sending all the required data to the GPU will be slower than doing it directly in the CPU.

We need to create a TileDrawOperation from a TileRequest. A TileDrawOperation has the concrete data required to draw the pixels of the Tile. In my case this is a list of a maximum of 4 elements, with the required "layers" of the final composition. We have at most one layer for each neighbor. (Again, in my current implementation I use 9 cells instead of 4, but I plan to change it in the near future).

First we need to know which tile "piece" from the original source tilemap (The ones I drawn manually) will be used for each neighbour.

To do so, I have a enum used as bitfield:

[System.Flags]
public enum TileMapPieceFlags : byte
{
    None = 0,
    SouthEast = 1 << 0,
    SouthWest = 1 << 1,
    NorthEast = 1 << 2,
    NorthWest = 1 << 3,
    All = SouthEast | SouthWest | NorthEast | NorthWest,
}

And then for each of the 4 neighbours, I construct a TileMapPieceFlags. This is called 4 times, one for each of the 4 neighbours:

private static TileMapPieceFlags GetFlags(TileSet current, QuadrantTileSets quadrant)
{
    TileMapPieceFlags tileFlags = TileMapPieceFlags.None;

    if (current == quadrant.TopLeft || current.Priority < quadrant.TopLeft.Priority) 
        tileFlags |= TileMapPieceFlags.NorthWest;
    if (current == quadrant.TopRight || current.Priority < quadrant.TopRight.Priority) 
        tileFlags |= TileMapPieceFlags.NorthEast;
    if (current == quadrant.BottomLeft || current.Priority < quadrant.BottomLeft.Priority) 
        tileFlags |= TileMapPieceFlags.SouthWest;
    if (current == quadrant.BottomRight || current.Priority < quadrant.BottomRight.Priority) 
        tileFlags |= TileMapPieceFlags.SouthEast;

    return tileFlags;
}

cuadrant is simply a struct that contains the TileSet for each of the 4 neighbour Covers. And current is the current TileSet of the four.

A think I haven't talked about the TileSet.Priority. Each TileSet has a priority that indicates if that tile layer will be above or below other pieces. A bigger priority indicates that the layer will be above, and a smaller priority indicates that the layer will be below. The priority must be manually assigned by me acording to my personal taste.

Here is an example with Grass with Priority=2 and Sand with priority=1. The grass covers the sand.

And here is the same with Grass with Priority=1 and Sand with priority=2. The sand covers the grass.

Remember that the original source tilemap was following a binary order? Once I have the TileMapPieceFlags I can easily convert this bitfield to a X,Y coordinate inside the tilemap:

// tiles are in a grid of 4x4 and has a size of TileSize x TileSize. 
int index = (int)piece & 0x0F; // piece is our TileMapPieceFlags
int x = index % 4;
int y = index / 4;
return new Vector2Int(x * TILE_SIZE, y * TILE_SIZE);

Once I have the 4 layers, each one with the TileSet of that neighbour, the priority of that TileSet and the coordinates to sample from the source tilemap, I need to add the layers. I create the 4 layers objects with that info and then sort them acording to the prioririty.

var layers = new TileDrawLayer[]
{
    GetLayer(quadrant.TopLeft, quadrant),
    GetLayer(quadrant.TopRight, quadrant),
    GetLayer(quadrant.BottomLeft, quadrant),
    GetLayer(quadrant.BottomRight, quadrant),
};

// Sort all layers in ascending order of priority
Array.Sort(layers, (a, b) => a.TileMap.Priority.CompareTo(b.TileMap.Priority));

foreach (var layer in layers)
{
    layerList.AddLayer(layer); // The Add method will remove the duplicate layers
}

Inside the AddLayer method I simply remove every layer that is empty or is repeated:

public void AddLayer(TileDrawLayer piece)
{
    if (piece.TileMapFlags == TileMapPieceFlags.None) return;

    if (this._layers.Count == 0)
    {
        this._layers.Add(piece);
    }
    else if (this._layers[this._layers.Count - 1].TileMap != piece.TileMap)
    {
        this._layers.Add(piece);
    }
}

That's all. After that we will have between 1 and 4 layers inside a list, all ordered by their priority. Inside each layer object will be the TileSet, and the source x,y coordinates to sample.

Drawing the layers with pixel art shadows

The final part of the process is to finally draw the layers in the image that will be used as our texture.

The idea is simple, for each pixel of the tile to draw, loop over all the layers and each pixel on top. If we did everithing right, we will get something like this:

It's not bad, but the problem with this simple method is that in real hand crafted pixel art, we normally use darker colours to shadow the areas. This generates a sort of "ambient occlusion" effect.

For example, take a look at this hand crafted pixel art tilemap:

"RPG Tiles: Cobble stone paths & town objects" by Zabin (Licence: CC-BY-SA 3.0) in OpenGameArt.org

If you look in the areas where two different materials intersect you will see how the top material (grass) produces a shadow over the material below (mud). Furthermore, the colours used to shadow the pixels are not shades of black, they are a pallete of colours based on the material below (mud).

So, we need to generate a palette of colours to use a shading colours. We could generate this palette manually for each tileset, or generate it automatically. In my case I generate them automatically when the tileset is loaded:

private static List<Rgba32> MakeLut(Image<Rgba32> texture, Rectangle rect)
{
    // rect is the rectangle coordinates of the tile piece that is "full", this is the last piece in my case. 

    // First, I sample ALL the colours in the piece and save them in a set.
    // I also calculate the average grayscale value (from 0.0f to 1.0f)
    var set = new HashSet<Rgba32>();
    float sum = 0f;
    for (int y = 0; y < rect.Width; y++)
    {
        for (int x = 0; x < rect.Width; x++)
        {
            var col = texture[rect.X + x, rect.Y + y];
            sum += ToGrayscale(col);
            set.Add(col);
        }
    }
    float avg = sum / (rect.Width * rect.Height);

    // After this, I have a set with all the palette used for the tileset piece.
    // I filter out all colors that are brighter than the average, and I only left the darker colors.
    // Then I order them by their brightness (from dark to bright)
    var list = set
            .Where(col => ToGrayscale(col) < avg)
            .OrderBy((col) => ToGrayscale(col))
            .ToList();

    if (list.Count >= 1)
    {
        // I take the darkest color and prepend it to the list but with an alpha of 50% so it mixes with the colour of the tilemap below. This method generates a nice transition.
        var extraColor = list[0]; 
        extraColor.A = 128;
        return list.Prepend(extraColor).ToList();
    }
    else
    {
        return list;
    }
}

That's the way I construct my colour palettes for shadow generation. But you can experiment tweaking those values or generating your palettes by hand.

After that, the process for shadowing a pixel is easy. If the pixel in the source image is pure magenta rgb(255,0,255) then I use the alpha component as a shadow percentage.

private static Rgba32 GetPixel(Vector2Int coord, TileDrawLayer layer, ref TileDrawLayer prevLayer)
{
    Rgba32 col = Sample(layer, coord);

    // The alpha channel is used as LUT index
    if (col.R == 255 && col.G == 0 && col.B == 255) // Pure magenta color
    {
        var shadeColor = prevLayer.TileMap.GetShadeColor(col.A / 255f);
        if (shadeColor.A == 255) return shadeColor;

        var layerDarkShadeColor = layer.TileMap.GetShadeColor(1f);
        col = Blend(layerDarkShadeColor, shadeColor);
    }
    else
    {
        prevLayer = layer;
    }

    return col;
}

The GetShadeColor simply is a lookup inside the LUT (Look up table) we previously created:

public Rgba32 GetShadeColor(float shadePercent)
{
    return this._lut[(int)MathF.Round((this._lut.Count - 1) * shadePercent)];
}

After this process, we will get this something like this:

I intentionally made the effect very subtle but you can play with your LUT generation to make a stronger shadow. Here is a comparison with and without shadow:

Summary

My tips are:

• Use "Visual tiles" instead of "Logical/Gameplay tiles" to reduce the number of required combinations.
• Organize your source tileset pieces using a binary bitfield so it's easier to convert find the required tile piece.
• Use a shadow color palette per tileset, either created by hand or generated automatically based on the tileset image.
• Use this palette to shadow the colors of the layers below the current layer.

I hope this post with my experience gives the required knowledge to others to improve the research in procedural pixel art texture generation. I couldn't find a lot of material in this area so it would be great to have more people interested in this.