DEV Community

How we use "ship small" to rapidly build new features at GitHub

Mike Coutermarsh on January 01, 2020

"Ship small! fast!" and "ship to learn!" You've seen that shouted everywhere on Twitter. But what does it actually look like in practice? I'm on ...
Collapse
 
emmanuelnk profile image
Emmanuel K

Great post! I have a question though that maybe wasn't addressed clearly. When you say 'ship small' I take it to also mean the ability to break the feature down into smaller logical parts and make small pull requests that can get reviewed faster and merged into production.

I constantly run into this situation where I sometimes receive/make massive pull requests because, well, all the code written was needed to make the feature functional (or makes logical sense together).

Is this a situation you encounter and if so how do you prevent it? Do you manage to keep all your PRs small enough to make the review process better for everyone?

Collapse
 
mscccc profile image
Mike Coutermarsh

Is this a situation you encounter and if so how do you prevent it? Do you manage to keep all your PRs small enough to make the review process better for everyone?

Definitely! Happens all the time and I still do this occasionally myself.

I have a few ideas around this.

  • Having feature flags helps out with this a lot. Because then the feature doesn't have to be functional to ship it.
  • A lot of this is cultural, the team has to be OK with merging things that are incomplete (for PR's like this, I recommend including an "up next" section in the description that lists out the plans to make it functioning)
  • If you find you have a huge branch. Sometimes I'll create a new branch, and then cherry-pick in specific files (just the backend for example). Put that in a PR to get it reviewed. Once it's merged, then I'll create another branch and cherry-pick in the frontend.

Also - sometimes making the PR small is more effort than it's worth. So sometimes it's necessary to have people review a large PR. I def try to avoid it though if possible.

Collapse
 
miku86 profile image
miku86

Hey Mike, thanks for your great post.

What's small for you? Do we talk about a PR per day? Per 3 days? 1 Week? Or is it more about the amount of code?

Sometimes there is a lot of code to write for a very small thing.
Any guidelines on this?

Thread Thread
 
mscccc profile image
Mike Coutermarsh

Hey!

For me, it's generally a day or two of work for a PR. I like to think about the person who is going to be reviewing it. Will it be easy for them?

Using feature flags and being OK with shipping partially working features helps make everything smaller.

I think it's OK to ship an empty page. Or a page that has static fake data, that needs to be connected to a database later. As long as it's all behind a flag and users don't see it, it helps the development process move faster.

Thread Thread
 
miku86 profile image
miku86

Thanks a lot,
I will keep that in mind!

Collapse
 
kpollich profile image
Kyle Pollich

Thanks for this, Mike! Really enjoyed this post. I have one question about the "parallel work streams" you mentioned.

  • 1 engineer can start working on the backend that feeds data to this page.
  • Another engineer can start working on the frontend.
  • The designer works on the UI. If they can code, they jump in and write markup + CSS.

What does the frontend work look like while the UI design is still being worked on? Is this like a "wiring up" of the initial boilerplate/wireframe that gets put up?

Thanks again!

Collapse
 
mscccc profile image
Mike Coutermarsh

❤️✨

Depends on how design heavy the feature is. Usually we have low fidelity sketches done during planning. Then we have our own design system, Primer, that we can copy/paste components from to build out the page. We can get really far with that alone.

This page for example: github.com/actions/starter-workflo...

I remember when we started it, we had some rough design ideas. We were confident we'd be showing "boxes" with workflows in them. So one engineer got started by passing an array of fake data to the frontend, and then looping over it to render a partial for each item in the array.

As this was happening, the designer was iterating on what the final page should look like. Once they had that, we were able to go in and adjust that partial/layout to look how the designer wanted.

My recommendation for people would be to always start on the piece you feel you have enough information to be confident in implementing (even if it looks terrible first pass). Just getting fake data on the page can sometimes be more work than we anticipate, so getting that out of the way and working is usually a nice win and saves time later.

Collapse
 
brpaz profile image
Bruno Paz

Great article!

What about improvements to existing features? Do you also use feature flags the same way?

And regarding Pull requests, if you work incrementally doing very small changes, while each PR might be much easier to digest, there will be many of them per day.

Can you talk a bit about how is your code review process is organized so that the amount of PRs waiting for review doesn't become a bottleneck?

Thank you! ;)

Collapse
 
mscccc profile image
Mike Coutermarsh

What about improvements to existing features? Do you also use feature flags the same way?

Yes, we often do. It really depends on the risk and is a judgement call for the team.

A recent example, we added this "filter bar" to the Actions tab.

We feature flagged this one because it can generate complex queries and the risk is it could cause performance problems when used against huge data sets. We also wanted to be able to flag in specific people and have them test the functionality before rolling out to everyone.

Can you talk a bit about how is your code review process is organized so that the amount of PRs waiting for review doesn't become a bottleneck?

Yup! A tip I learned from @andreasklinger a few years ago. When you work on a remote team you need everyone to dedicate time to doing code reviews. Unblocking each other is a priority, so most people will dedicate time in the AM (or whenever works for them) each day and they'll go through all the PRs and review them.

I find that when starting these projects, usually the first few PRs are a little more high effort to review because we're setting the foundation for the feature. But after that, things are small incremental improvements to the base. Since people are familiar with the area of code, they get pretty quick to review.

We rarely see anything go unreviewed for more than 24 hours. It helps that we are all on the same project and are motivated to get it done. If someone started a PR, but is now outside of their working ours, we'll often have someone else push commits to it to finish it up and ship it. Seeing multiple peoples commits on a single PR is pretty common.

One more thought on being "unblocking". We almost never block a PR unless we see something in the code that would cause a huge problem. We'll approve it, and leave "non-blocking notes" for things to improve. Then it's up to the submitter to take that feedback and apply it if they want. This reduces the amount of back and forth (critically important when in different timezones).

Collapse
 
brpaz profile image
Bruno Paz • Edited

Thanks for the response. Perfectly clear ;)

Just one more question about feature flags.

How do you manage the cleanup process of older feature flags that are no longer needed, for example after the feature is generally available in production?

Thread Thread
 
mscccc profile image
Mike Coutermarsh

We're not great at that :). We often leave them in code for a couple months. Then we file issues to remove them and people will pick those up when they're looking for a quick/easy task to do.

Collapse
 
mgohin profile image
Maelig

Thanks for your sharing, it gives me hope that one day, I'll work in teams like that, maybe in 20 years :)

Got a question about production visibility, can you describe (or maybe you did talk about it elsewhere, or will) how works your "feature flags", because it's the main advantage you have to push code to production without giving access to "real" users.

Thanks ;)

Collapse
 
mscccc profile image
Mike Coutermarsh

:)

We use Flipper + some custom UI built internally. It works for Ruby apps.

If you're not using Ruby, I recommend looking at launchdarkly.com.

Collapse
 
tonivdv profile image
Toni Van de Voorde

Great post and this is exactly how we are working at Adsdaq too. Since we introduced feature flags we not only can deliver/deploy fast but we completely eliminated our "long-living" branches which are a real pain.

There is one thing not covered here though which is very important to know when starting this direction and that is "backwards compatibility" ... Sometimes you need to change an existing feature which impacts something either in the api or the database. Since you are deploying and hiding with a feature flag, changes to those things must make sure the existing stuff keeps working! This can sometimes be very challenging. But not doing it immediately breaks the "deploy quickly and often" paradigm.

Collapse
 
dabeeeenster profile image
Ben Rometsch

How are you guys managing your feature flags? We Open Sourced our Feature Flag platform Bullet Train last year and would love any feedback you have.

Do you think its better to roll your own feature flag system or rely on a third party?

Collapse
 
mscccc profile image
Mike Coutermarsh

Haven't heard of Bullet Train, looks nice!

We use Flipper, which is open source: github.com/jnunemaker/flipper. We have a custom UI on top of it for administration.

I'd always use something open source or paid. Building one doesn't seem like a good use of time when there are so many great OSS ones out there.

Collapse
 
yo profile image
Yoginth

I found Unleash - Open source enterprise ready feature toggle service

unleash.github.io

Collapse
 
serhiiohorodnyk profile image
Serhii Ohorodnyk

Really good post! Thank you for your effort.
I have a question regarding feature flags in the backend. What if your new feature requires relatively big or significant database migrations. How does that impact the process? Do you maintain two data structures (old and new) till the feature is done?

Collapse
 
mscccc profile image
Mike Coutermarsh

That really depends on the migration + feature.

We can never take downtime at GitHub. So when making any DB change, it has to work with both the old and new code.

A recent example, we just had a feature that required us to move data to a new table. The process was like this.

  • Create the new table (no code changes)
  • Change writes to both the old and new table
  • Update the new feature to read from the new table (feature is behind a flag, so OK if data is incomplete)
  • Run backfill, transferring old data to the new table (slowly over a week)
  • Change old feature to read from the new table
Collapse
 
abdurrahmaanj profile image
Abdur-Rahmaan Janhangeer

Awesome Read!