Back in January Jeff Hodges wrote a wonderful essay: “Notes on Distributed Systems for Young Bloods.”  There is a lot of good stuff in there.  I was remembering this recently because this section highlights something I’d not thought so clearly about; i.e. that feature flags run contrary to the usual software engineering best practice – as explained in the next to the last paragraph of this snippet:

“Feature flags are how infrastructure is rolled out. “Feature flags” are a common way product engineers roll out new features in a system. Feature flags are typically associated with frontend A/B testing where they are used to show a new design or feature to only some of the userbase. But they are a powerful way of replacing infrastructure as well.

Suppose you’re going from a single database to a service that hides the details of a new storage solution. Have the service wrap around the legacy storage, and ramp up writes to it slowly. With backfilling, comparison checks on read (another feature flag), and then slow ramp up of reads (yet another flag), you will have much more confidence and fewer disasters. Too many projects have failed because they went for the “big cutover” or a series of “big cutovers” that were then forced into rollbacks by bugs found too late.

Feature flags sound like a terrible mess of conditionals to a classically trained object-oriented developer or a new engineer with well-intentioned training. And the use of feature flags means accepting that having multiple versions of infrastructure and data is a norm, not an rarity. This is a deep lesson. What works well for single-machine systems sometimes falters in the face of distributed problems.

Feature flags are best understood as a trade-off, trading local complexity (in the code, in one system) for global simplicity and resilience.”

I think that helps to explain why the things that get built out over time so often seem to be in such desperate need of a total rewrite.  The litter left behind by the search (a/b testing) and the vistigal organs of the switch overs all frustrate some people’s sense of good design.  The design patterns at one scale or dimension are at odds here those of another.