Problem
In 2020 like most people our company transition to more remote work and I started to invest in my home office to prepare for the coming onslaught of zoom meetings. Once I had a comfortable setup, I had everything I needed to start streaming outside of work as well because why not right? I started playing around with twitch and cutting my videos down for youtube.
Most of my streams were 2+ hours and while playing 2+ hours of games with your friends is an investment in relaxation and friendship. Watching back 2+ hours wasn't a time investment I wanted to make.
Conception
My game of choice is Valorant at the moment but this is true of most games; In the game there is a visual queue to show you have scored a point.
Because they are UI elements they are usually visually bright, consistently placed, and distinct from the background view of the game.
The score indicator is kind of like a flashing light, if I had something like a Light-Dependent Resistor, I could record when it flashed.
Application
I have a concept "check for increases in white in the video"
MDN has a great example of how to check each frame of a video using HTML canvas. The example is set up like this:
ctx1 = Canvas 1 context
ctx2 = Canvas 2 context
We are going to borrow their function and focusing on the computeFrame section. You can see below the RGB values for each pixel in each frame.
processor.computeFrame = function computeFrame() {
//drawing the full frame to canvas
this.ctx1.drawImage(this.video, 0, 0, this.width, this.height);
//get the frame from canvas at 0 x and 0 y
let frame = this.ctx1.getImageData(0, 0, this.width, this.height);
let l = frame.data.length / 4;
for (let i = 0; i < l; i++) {
let r = frame.data[i * 4 + 0];
let g = frame.data[i * 4 + 1];
let b = frame.data[i * 4 + 2];
if (g > 100 && r > 100 && b < 43)
frame.data[i * 4 + 3] = 0;
}
this.ctx2.putImageData(frame, 0, 0);
return;
}
In the example, above it's checking this range of color and if it falls within the threshold it will make it alpha instead thus creating a green screen or in this case yellow screen.
Simple enough, i will just check for white pixel in the area.
//255,255,255 is white so 240 -> 255 is mostly white
if (g > 240 && r > 240 && b < 240)
//is white pixel
}
But the game has complex visuals and many elements would trigger just "white"
Every picture is made up of an almost unique amount of colors and shades so all I needed to do is get as close as possible to that unique number.
let skullFound = []
let white = []
let green = []
let red = []
for (let i = 0; i < l; i++) {
let r = frame.data[i * 4 + 0];
let g = frame.data[i * 4 + 1];
let b = frame.data[i * 4 + 2];
if (isWhite(r,g,b))
white.push({r,g,b})
}
if (isGreen(r,g,b))
green.push({r,g,b})
}
if (isRed(r,g,b))
red.push({r,g,b})
}
}
if(whiteThreshold(white.length) && greenThreshold(green.length) && redThreshold(red.length)) {
skullFound.push(video.currentTime)
white = []
green = []
red = []
}
After 30ish minutes of trial and error, I was able to get about a 99% accuracy rate at 2x speed with the videos tested with 1 main exception being, if the Character Sage is within the cropped section when she is shot with a sniper rifle... which is kinda rare.
Conclusion
While the current system is not perfect, It's a simplistic solution to the problem I was facing that can be built upon later.
I believe methods like the one above can be applied to many game videos. I look forward to finding more fun techniques in this area in the future.
Top comments (3)
this is cool, one question:
does it scan the video at play speed? or is it pissible to scan faster? maybe only one in 10 frames need to get analyzed.
Sorry I didn't include it in the Article but yes It scans at 2x speed. I tried higher speeds or less frames but because the element is animated the colors vary based on the frame of the skull animation recognition went down. I did play with skipping frames but again if you miss the most accurately matched frame, it might not register. I am definitely looking at this because the faster the process the better.
nice, thanks, i understand that there is a trade off.