How I made Flappy Pose, Flappy Bird but where you flap with your arms

Oz Ramos

@checkboxoz

Day 5: I made a Flappy Bird but where you have to flap your arms! Try it on @Glitch: flappy-pose.glitch.me

Click the fish at top right to view the source or remix it! Made with @p5xjs and @ml5js #100DaysofCode

04:38 AM - 29 Feb 2020

56 238

Now that my muscles aren't sore from a day full of flapping, I wanted to take a moment to show you how I made the flapping portion of the game. For brevity, I'll only be covering the flapping portion, but if you'd like to see how I made the actual game itself I recommend watching this video:

Initializing PoseNet through ml5.js

ml5.js is a super friendly library for using Machine Learning on the web. It comes packaged with a few computer vision models, and the one we're interested in for flapping is PoseNet.

So the first thing I did was add ml5.js and p5.js to the page:

<script src="https://unpkg.com/ml5@0.4.3/dist/ml5.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/p5@0.10.2/lib/p5.js"></script>

The next step is to initialize PoseNet. This is done by passing a video element, a callback to run once it's done, and optimization options:

  // createCapture is a P5.js method that creates a webcam stream
  video = createCapture(VIDEO)
  video.hide()

  // Initialize posenet...easy peasy!
  posenet = ml5.poseNet(video, () => {console.log('posenet ready!'}, { maxPoseDetections: 1})

The next step is to watch for flaps. In my approach, I consider a flap to start when the wrists go above shoulder level and end when the wrists fall back below shoulder level.

To determine how this happens, we first need to listen for posenet poses. This is done with posenet.on('pose', callback). The callback here receives a pose object with 17 keypoints and their positions. The ones we're interested in are keypoints 9 and 10 (wrists) and 5 and 6 (shoulders).

To get flapping to work, we simply compare the keypoints y values and run a method when the flap occurs:

  posenet.on('pose', poses => {
    pose = poses[0].pose

    // Detect up flap
    // (Note: I add 40 to the y so we don't have to flap as high)
    if (
      pose.keypoints[9].position.y < pose.keypoints[5].position.y + 40 &&
      pose.keypoints[10].position.y < pose.keypoints[6].position.y + 40
    ) {
      hasFlappedUp = true
    }

    // Detect down flap
    if (
      hasFlappedUp &&
      pose.keypoints[9].position.y > pose.keypoints[5].position.y + 40 &&
      pose.keypoints[10].position.y > pose.keypoints[6].position.y + 40
    ) {
      hasFlappedUp = false

      // FLAP 🐦
      bird.flap()
    }
  })

That's all there is to it!

You can look through the source code on Glitch, or try the demo out yourself.

Thanks for reading and happy flapping!

Follow me on Twitter @CheckboxOz