DeathMark: Programmatically scan videos for points of interest

Problem

In 2020 like most people our company transition to more remote work and I started to invest in my home office to prepare for the coming onslaught of zoom meetings. Once I had a comfortable setup, I had everything I needed to start streaming outside of work as well because why not right? I started playing around with twitch and cutting my videos down for youtube.

Most of my streams were 2+ hours and while playing 2+ hours of games with your friends is an investment in relaxation and friendship. Watching back 2+ hours wasn't a time investment I wanted to make.

Conception

My game of choice is Valorant at the moment but this is true of most games; In the game there is a visual queue to show you have scored a point.

Because they are UI elements they are usually visually bright, consistently placed, and distinct from the background view of the game.

The score indicator is kind of like a flashing light, if I had something like a Light-Dependent Resistor, I could record when it flashed.

Application

I have a concept "check for increases in white in the video"
MDN has a great example of how to check each frame of a video using HTML canvas. The example is set up like this:

ctx1 = Canvas 1 context
ctx2 = Canvas 2 context

We are going to borrow their function and focusing on the computeFrame section. You can see below the RGB values for each pixel in each frame.

 processor.computeFrame = function computeFrame() {
//drawing the full frame to canvas
    this.ctx1.drawImage(this.video, 0, 0, this.width, this.height);
//get the frame from canvas at 0 x and 0 y 
    let frame = this.ctx1.getImageData(0, 0, this.width, this.height);

    let l = frame.data.length / 4;

    for (let i = 0; i < l; i++) {
      let r = frame.data[i * 4 + 0];
      let g = frame.data[i * 4 + 1];
      let b = frame.data[i * 4 + 2];
      if (g > 100 && r > 100 && b < 43)
        frame.data[i * 4 + 3] = 0;
    }
    this.ctx2.putImageData(frame, 0, 0);
    return;
  }

In the example, above it's checking this range of color and if it falls within the threshold it will make it alpha instead thus creating a green screen or in this case yellow screen.

Simple enough, i will just check for white pixel in the area.

//255,255,255 is white so 240 -> 255 is mostly white
 if (g > 240 && r > 240 && b < 240)
        //is white pixel
    }

But the game has complex visuals and many elements would trigger just "white"

Every picture is made up of an almost unique amount of colors and shades so all I needed to do is get as close as possible to that unique number.

let skullFound = []
let white = []
let green = []
let red = []
for (let i = 0; i < l; i++) {
      let r = frame.data[i * 4 + 0];
      let g = frame.data[i * 4 + 1];
      let b = frame.data[i * 4 + 2];
       if (isWhite(r,g,b))
         white.push({r,g,b})
       }
      if (isGreen(r,g,b))
        green.push({r,g,b})
      }
      if (isRed(r,g,b))
        red.push({r,g,b})
      }
}

if(whiteThreshold(white.length) && greenThreshold(green.length) && redThreshold(red.length)) {
 skullFound.push(video.currentTime)
 white = []
 green = []
 red = []
}

After 30ish minutes of trial and error, I was able to get about a 99% accuracy rate at 2x speed with the videos tested with 1 main exception being, if the Character Sage is within the cropped section when she is shot with a sniper rifle... which is kinda rare.

Conclusion

While the current system is not perfect, It's a simplistic solution to the problem I was facing that can be built upon later.

I believe methods like the one above can be applied to many game videos. I look forward to finding more fun techniques in this area in the future.

Free Download

https://ko-fi.com/post/DeathMark-W7W8381IO

Comments

[Tobias Nickel]

this is cool, one question:

does it scan the video at play speed? or is it pissible to scan faster? maybe only one in 10 frames need to get analyzed.

[Alex King]

Sorry I didn't include it in the Article but yes It scans at 2x speed. I tried higher speeds or less frames but because the element is animated the colors vary based on the frame of the skull animation recognition went down. I did play with skipping frames but again if you miss the most accurately matched frame, it might not register. I am definitely looking at this because the faster the process the better.

[Tobias Nickel]

nice, thanks, i understand that there is a trade off.