Why Feature Flags Should Be Treated as Production Infrastructure

Beginning with a very simple concept - to turn on/off new functionality using a switch - many developers use feature flags to prevent needing to deploy the application again.The initial implementation may seem harmless at this stage (see below):if (featureFlags.newCheckout) { return ; } return ; However, as the product evolves, there will be more and more parts of the system controlled by feature flags, including but not limited to: checkout flow, pricing experimentation, onboarding experience, migrating APIs, changing navigation, adding payment methods, and emergency shutdown mechanisms.Once you reach this level, they are no longer simply conditional statements within your codebase. The feature flag is now part of your application's production infrastructure.Thus, treating them as such will significantly alter how development teams design, operate, monitor, and eventually remove these switches.Feature Flags Change Production BehaviorChanging the codebase to what's in production changes what code exists in production. Changing the way production behaves (behind a feature flag) changes which version of production behavior users see.This difference is crucial.Suppose you have a team that deploys a brand new check-out flow (which they did behind a disabled flag). Although technically, the deployment was successful, your customers were still using the old experience. The team could turn on the new check-out for employees, and then 5% of customers, and then 25%, and finally all customers.At each stage of this process, there will be different application behavior without requiring another deployment.Therefore, changing a flag to turn on or off a feature can sometimes require the same level of operational risk as making a code change. There is no difference if someone turns on an incomplete payment flow for 100% of your customers and no deployment happened; it doesn't matter.Therefore, production flags need proper controls.Flags Need Clear OwnershipOne of the main issues with using feature flags is that it often causes confusion about who owns them and what they are intended to do.An example of how this works is a team developing a flag for a new feature; after the feature has been released without incident and all attention is turned toward development of the next project, the flag remains in the codebase for approximately two years. As time passes, no one seems to know if it's safe to remove the flag from the code.Each production flag should have both a designated owner and a defined use. Teams should also be able to answer additional questions regarding each flag, such as: Who is the owner of this flag? For what reason was this flag created? What will occur once this flag is activated? Will the activation of this flag be a temporary measure, or will it remain active indefinitely? At what point would you plan on removing this flag?If there is no defined owner for your flags, then they will slowly develop into technical debt.Rollouts Should Be GradualOne of the greatest benefits of feature flags is that you are able to separate deployment (the process of making changes available to customers) from release (when those changes become active for all users). As opposed to deploying a large change to every single user as soon as it has been made, teams can deploy a new change in phases.For example:Internal users → 1% → 10% → 25% → 50% → 100%Teams can measure errors, latency, conversion rates, support requests, and many other key performance indicators during each phase. If anything goes wrong at 10% — even if that problem would not have been experienced by enough customers to be considered widespread at this point — the rollout can stop right away so that there will never be enough affected customers to create a broader concern. This makes feature flags an important risk-management mechanism. They provide a practical middle ground between "not released" and "released to everyone."Feature Flags Need ObservabilityA flag without monitoring provides limited protection. Suppose a team enables a new search experience for 20% of users. Error rates increase shortly afterward. Engineers need to know whether those errors are coming from users receiving the new experience or from unrelated traffic. Logging and analytics should therefore include relevant feature flag context.For example, an error event might contain:feature: new-searchvariant: enableduser_cohort: rollout-20Now engineers can compare behavior between enabled and disabled populations. The same principle applies to performance monitoring and product analytics. If a feature is controlled by a flag, teams should be able to understand how that flag affects the system. Otherwise, gradual rollout becomes little more than gradual guessing.Kill Switches Need Special TreatmentSome flags were put into place for safety reasons (to protect users or applications). Consider an application that adds in a third-party recommendation system. If this recommendation provider becomes very slow, it may bring down the whole page because of the core application's dependency on it.The kill switch allows developers to turn off recommendations immediately while leaving their main application running. These are different than typical experimentation flags that teams use for testing. Teams need to know:Exactly what will happen if the kill switch is turned on.Who will be able to turn on the kill switch.Will the fallback path work?An untested kill switch will give your team a false sense of security.Permissions and Audit History MatterAs feature flags continue to grow in power, access control will become increasingly important. Not all developers should be allowed to turn on every single feature flag for each production user.For example, a flag that controls how buttons are styled is going to be dramatically different than a flag that controls payments or authentication.As such, a production flag system should have the ability to enforce proper permissions as well as keep track of which engineer has made changes to the flag, along with what was changed and when. This becomes extremely beneficial in incident scenarios.In those situations, instead of sending out a mass email to see if anyone has made recent changes, engineers can simply look at the historical log for the specific flag related to the issue and identify if there were any configuration changes to production during the same timeframe the problem began.Old Flags Must Be RemovedFeature flags provide flexibility; however, once a rollout has reached 100% and the new implementation is stable, the flag and legacy code paths should typically be eliminated to avoid accumulation in the codebase of irrelevant conditions.Otherwise, codebases accumulate conditions such as:if (flagA) { if (flagB) { // ... } } As time progresses, developers will have to reason about all possible state combinations, which could potentially be obsolete. Testing becomes harder, debugging becomes more difficult, and it can also be dangerous to remove existing functionality. Teams should conduct an ongoing review of their flag inventory for stale flags. Eliminating these flags can be viewed as another aspect of completing a feature versus being optional maintenance work.ConclusionAs a flag's usage grows in an organization, it will start to represent actual production behavior. The decisions made by flags include who sees what features, how tests for new features run, how gradual changes are rolled out, and how fast teams can react when things go wrong.This is why organizations with large engineering teams should consider treating their feature flags as part of their production infrastructure.Like all other pieces of their production infrastructure, feature flags require ownership, permissioning, monitoring, audit trails, rollback plans, and cleanup plans.While the primary benefit of using feature flags is that they provide teams with a way to turn features on and off, the true benefits of feature flags come from providing teams with a way to make changes to their production environment while controlling risk.