Using Feature Flags in Sentry

From testing in production to running A/B tests, feature flags have a range of uses. At Sentry, one way we use feature flags is to safely allow beta access to new features for some of our “Early Adopter” customers.

Because you can set multiple combinations of feature flags, every user is likely to have a different experience. While that's great for finding answers and iterating on product, it can be increasingly difficult to determine where certain errors and performance issues are coming from. Are my users having trouble because they are part of experiment A or experiment B — or some combination of the two?

Tagging feature flags

Thankfully, Sentry can answer these questions using tags. Tags are key/value string pairs that you can bind to your events and transactions.

You'll want to set a new tag for each feature flag you want to surface in Sentry. Here is some code in one of my apps that passes in the show_widget feature flag for a given user:

Sentry.init({
    dsn: '__DSN__',
    // ...
});

Sentry.setTag("show_widget", user.showWidget);

After you start passing in feature flag data, you'll begin seeing your new tags in Sentry. In the example below, you can see that this issue primarily impacts those users who are seeing our new widget feature.

Using Discover + Tags

Once you have your tags set up, you can use Discover to see which combination of feature flags are causing errors or performance issues.
In Sentry, navigate to Discover and click Build A New Query. Then, add columns for each of the feature flags/experiments in question.

Once you've done that, you'll be able to see which flags are causing problems for your users.

It looks as if errors were primarily impacting the intersection of users who have both the show widget feature enabled and variation 2 of our new checkout experiment. From here, we’re able to use Discover to drill down exactly what issue is causing problems. Then we can decide which feature flag needs to be disabled until we have had the chance to fix the problem. Crisis averted.

Feature flags allow you to control access for a variety of release scenarios: new features for early adopters, organizations with special plans, or even incremental releases. By safely activating — or deactivating — certain features, flags can close your feedback loop, while allowing you to address an issue without the pressure and stress of a live incident.

Want to learn more about what we’ve done with feature flags? Here’s our latest Dogfooding Chronicles post. If you’re new to Sentry, you can try it for free today.