Shadow Shader Experiments

Introduction

After playing around with light and normal mapping, the next thing that comes to mind is shadows. Instead implementing something from the shelf like shadow mapping or ray tracing, this post deals with a custom implementation. The implementation is based on an idea, it has (severe) limitations, it might not work, other people might have already implemented or refrained from implementing it because they had more foresight than me, ... I don't know, but we'll find out.

DISCLAIMER Again, this is just experimental stuff and likely a waste of time. If you value your time, stop reading now. If you proceed, don't say I haven't warned you. ;)

So, let's go and have some stupid fun! ;)

The Basic Idea

Given that I have a pixel art game with a low resolution (currently tending to 320x240 internal) and I already have some per-pixel lighting in place, the idea is as follows. For every pixel I'll check whether the neighboring pixel is elevated and in between the current pixel and the light source, and if so darken it a bit. One limitation that is a given is that my shadows can only be one pixel large.

We need eight values for every pixel that indicate whether each of the neighboring eight pixels is elevated or not. It's kind of a degenerated gradient descent map.

We might even be able to adjust the shadow based on the difference in angle between the light source and the pixel that casts shadow on the current pixel.

We already have the direction from the current fragment to the light source from the normal mapping post.

An Attempt at Implementation

We need eight boolean values for each pixel. Similar to using normal maps to store non-color information, another texture seems to be a good candidate for this. We can just use the bits of an 8bit color channel to encode this information. Let's define the pixel directly above as 0 and then go around clockwise.

+---+---+---+
| 7 | 0 | 1 |
+---+---+---+
| 6 | x | 2 |
+---+---+---+
| 5 | 4 | 3 |
+---+---+---+

0: 1000'0000
1: 0100'0000
2: 0010'0000
3: 0001'0000
4: 0000'1000
5: 0000'0100
6: 0000'0010
7: 0000'0001

if the pixels 0, 1 and 2 were elevated in comparison to x, we would encode the value 1110'0000.

To make it easier to draw the gradient map, we convert these binary values to hex values. b1110'0000 becomes 0xE0 for example. We'll use the red channel, therefore the entire color's hex values is 0xE00000. We could create a small tool to make this process easier, but since it's not clear whether this approach has any future, we just do everything by hand.

We have a 3x3 grid which leads to quite a few permutations, but we don't need all of them.

At the top of the image is the final gradient map for the floating rock. Most of the image is black, which means there's no difference in elevation. Below are some of the possible variations from which we can pick the corresponding color.

Now that we have the texture we'll need to do something with it in the fragment shader. First we need to a add another texture sampler, from which we'll then get the red channel. We can check whether a certain direction is above the current fragment by using the bitwise AND operator & (details in the shader comments).

// ...
uniform sampler2D gradientMap;
// ...

void main() {
    // ...
    // Get the red channel from the gradient map, this is where our elevation information is stored.
    float redChannel = texture(gradientMap, TexCoord).r;
    // Convert it to an integer as bit-wise operators only work with integers
    int intValue = int(redChannel * 255.0);
    // Check for each direction if it's elevated (1) compared to our current position.
    int north = intValue & 0x80;
    int northEast = intValue & 0x40;
    int east = intValue & 0x20;
    int southEast = intValue & 0x10;
    int south = intValue & 0x08;
    int southWest = intValue & 0x04;
    int west = intValue & 0x02;
    int northWest = intValue & 0x01;

    // Next, calculate the shadow by adding a value for every direction whose pixel is elevated. The values for the variables north, northEast, ... are all greater 0 if they are elevated and should therefore add to the shadow value.  The step function returns 0 if the value of the second parameter is smaller than the first one, and 1.0 otherwise. This way only the directions > 0 contribute to the shadow value and all to the same degree.
    float shadow = step(0.1f, north) * max(dot(vec3(0, 1, 0), lightDir), 0);
    shadow += step(0.1f, northEast) * max(dot(vec3(1, 1, 0), lightDir), 0);
    shadow += step(0.1f, east) * max(dot(vec3(1, 0, 0), lightDir), 0);
    shadow += step(0.1f, southEast) * max(dot(vec3(1, -1, 0), lightDir), 0);
    shadow += step(0.1f, south) * max(dot(vec3(0, -1, 0), lightDir), 0);
    shadow += step(0.1f, southWest) * max(dot(vec3(-1, -1, 0), lightDir), 0);
    shadow += step(0.1f, west) * max(dot(vec3(-1, 0, 0), lightDir), 0);
    shadow += step(0.1f, northWest) * max(dot(vec3(-1, 1, 0), lightDir), 0);

    // limit the value to a range of 0 to 0.5, choose whatever range looks good to you
    shadow = clamp(shadow, 0, 0.5);
    // Since we subtract the value later on, we need to set the alpha channel to 0.0f. If we'd set it to a greater value we'd influence the transparency of the final fragment color.
    vec4 shadowVec = vec4(shadow, shadow, shadow, 0.0f);
    FragColor = texture(tex, TexCoord) * (ambient + diffuseColor - shadowVec);
}

Let's have a look at how it looks like.

The little dot about the green stuff is the light source.

It's nothing to phone home about, but it's not that bad either.

The Struggles

I wasn't sure whether I should implement it and even more so whether I should blog about it. Also, there were multiple times when I wanted to optimize for performance and had to remind myself that this is just a proof of concept and now was not the time to do that. I might have wasted some time for a solution that I'll likely not use in the future. On the other hand I've learned some things and have gotten more comfortable with shaders. Trying to figure out something from the ground up, without just following a tutorial or checking what other options are out there. Aaaand it was fun.

Implementation-wise I took a detour by using floatBitsToInt which just interprets the float bits as an int. Not what I was looking for and the solution was much easier - a simple int(x) cast.

Thanks for reading and keep on struggling.