I confess that “Feature Flags” make me a bit nervous. Despite this I think them a useful and important tool in our ability to achieve Continuous Integration.
So why do they make me nervous? Well, they are a form of “branching” they are designed to isolate change and as I have described previously, I think that branching works against Continuous Integration.
There is a big difference between Feature Flags and VCS-based branches though. Feature Flags isolate at the level of behaviour rather than at the level of code. This is an important and valuable distinction.
“Feature Flags isolate at the level of behaviour rather than at the level of code”
One of my practical objections to the use of VCS branches for normal development is that they place a barrier to promiscuous refactoring. All of the best code-bases that I have worked in had a high churn-rate. We would change them often and make them better in small ways all the time. Branches tend to prevent us from doing that, Feature Flags, on the other hand, allow it.
The danger with Feature Flags is that they can introduce considerable complexity. Which version of your code do your test? Feature on or off? Both?
If “both”? you are on a journey into exponential complexity growth as you add more flags – Flag “A” on Flag “B” on, Flag “A” on Flag “B” off, and so on! This is a never-ending game that you can’t win in any definitive way.
I tend to employ a hierarchy of approaches to allow me to make progressive changes in my code running under Continuous Integration.
First, I prefer to release change directly, make a change and have people use it.
Next, I will use “dark-release” or “branch-by-abstraction”. Dark-release allows me to build up, and test, stuff that people aren’t using yet. Branch-by-abstraction encourages me to create abstractions in my code. These abstractions allow me to switch the implementation of these abstract features easily. It also fits with my style of coding where I care very much about separation-of-concerns and abstraction. Branch-by-abstraction can even allow me to run the old and new versions of a feature in parallel! This opens another world of possibilities for measuring the merits, or otherwise, of new features.
Only if none of these work will I use Feature Flags. This is largely because of the testing problem. For the types of systems that I have worked on for the past few years, I want to test what is running in production.
Another facet of this is that, on the whole, I prefer to make my Deployment Pipeline so efficient, that if I want to change the config of my system, even its Feature Flags, I will push the change through the pipeline, and so I can test it before release!
Hi Dave! Looks like this post builds on the previous one, and as such reveals some of the same thinking traps/pitfalls.
The big one here is the following statement that mischaracterizes the difference between VCS-branches and feature-flags (conditional branches):
[FALSE] “Feature Flags isolate at the level of behaviour rather than at the level of code” [/FALSE]
In fact there are two big problems with this statement:
1. VCS-branches do NOT isolate/branch at the code-level. At best, they isolate/branch at the repository-level (i.e. the codebase) but still not along the physical code dimension. VCS-branches simply do not shape code-structure nor its execution.
2. Feature-flags do NOT isolate/branch at the level of behavior. Feature-flags isolate/branch precisely at the physical code level, and exactly effect code structure and execution.
The fact that both feature-branch and feature-toggle use the word “feature” is what indicates the intended granularity of what is being isolated (i.e., along behavioral boundaries).
Code-toggles quite literally branch/isolate the structure+execution of the code — in fact, you could even call them “branches as [conditional] code” (whereas VCS branches are “branches as [concurrent] versions”).
When talking about feature-flags vs feature-branches, they key differentiating factor (or dimension) is *binding-time*, which could be any of: version-selection-time (checkout/extract/clone), checkin/commit-time, compile-time, link-time, install/upgrade-time, run-time, etc.
This brings us to the next mischaracterized assumption: what precisely *needs* to be isolated? It certainly has something to do with the behavioral boundaries/scope of feature, but is it really the development work for the feature, the implementation code of the feature, or the execution-availability of the feature in a production-environment?
Many learned from the famous gang-of-four Design Patterns book to “isolate/encapsulate the thing that varies” (along with the other key lessons of “prefer composition over inheritance” and “program to an interface, not to an implementation). So which of these three aspects of the feature is it that we need to let vary in this case (code development-path, code implementation-structure, code execution-path)
This all comes down to managing commonality and variability, and at the proper *binding-time*! When choosing between a feature-branch or a feature-toggle, we have already made a decision that *something* has to be delayed (later) so it can be realized all-at-once instead of incrementally over time. And in our case we know we do NOT want that thing to be integration-feedback. So if it’s not integration-feedback, should it be delivery-feedback? deployment-feedback? Release-feedback? (the answer will tell us the proper binding-times for our selectable variation).
When we use a toggle in code, that is a conditional-branch that is executed/selected at a later binding-time (and I am assuming we don’t want to use conditional-compilation, because I prefer to call that a compile-flag or a build-flag rather than a feature-flag).
When we choose to use a feature-branch *or* a feature-flag, we aren’t choosing whether or not to delay integration-feedback. Either way, we need to be disciplined about integrating frequently to main-trunk throughout the lifetime of the feature’s implementation effort. The branch doesn’t prevent us from doing that, and neither does the toggle.
So what is causing the hesitation to integrate and build frequently? Is it the fear of being able to “back out” (rollback) the code-changes? Is it the fear of having to rework a bad design instead of refactoring it into place incrementally? Or is it the fear of delivering incomplete functionality (that works, but is less than the previously-specified scope)?
What is it that toggling lets us do incrementally versus all-at-once? And what is the risk and/or technical debt incurred by doing it that way?
Toggling is one way of letting us delay the selected binding-time between the developed code, and its execution in the production environment. The problem is toggling uses conditional code-statements rather than more involved design constructs/patterns. And many of the “code-smells” we normally want to refactor are of the form “replace conditional with *x*” where “x” is one of polymorphism, composition, delegation, concurrency/parallelism/distribution, or some other technique to achieve a greater “separation of concerns” between the form of variation we are trying to isolate, and what we are trying to isolate it from.
This is why feature-toggles are intended to be temporary or short-lived (just like *development* branches, and/or the time between integrations), because we don’t want to incur too much technical debt as a result of added code-complexity for merging/integrating NOR for refactoring or testing.
If the feature-toggle is too long-lived, it adds too much technical debt (“smelly code”) and creates more rework (not just for subsequent merging, for ANY subsequent changes). But if we want the toggle to be ling-lived, then we either have to do a more complex up-front design to apply one or more design-patterns to give a proper separation of concerns *OR* we need to make sure the toggles are NOT exempt from refactoring efforts (e.g. refactoring to patterns”) in order to keeping the code as simple, non-duplicated, and intention-revealing as possible.
The problem (or fear) with feature-toggles (as with branches) is that they will become too-long-lived *and* *also* that the necessary incremental changes & feedback wont happen continuously enough (whether it is continuous integration, continuous design [refactoring], continuous-testing [TDD/BDD/ATDD], continuous delivery, or continuous deployment).
This is also why I don’t like the false dichotomy between feature-branching XOR feature-toggling (assuming you need to replace one with the other) *and* the false correlation between existence of a version (or code) branch, and lack of discipline to break up integration (or design) into small, frequent, and continuous chunks.
whether or not you branch (and for how long) isn’t as critical as when (the binding time) and how frequently you integrate feedback and changes, and the former is NOT what dictates the latter (if anything, it is the other way around).
Hmmmmm, I don’t think that I have not been condescended to quite so much for a long time now. 😉
Again, I think that we will have to agree to disagree.
If you want to be pedantic then I should probably have said “source-code” rather than “code” but I stand by my assertion that branches isolate at the source-code level and toggles at the level of behaviour. If I want to make a change, a refactoring perhaps, in branched code that change will be deferred until the branches are merged. If the behaviour is isolated with a toggle the change will happen at the same time – you can see its implications in your IDE even before you commit!
df: “Hmmmmm, I don’t think that I have not been condescended to quite so much for a long time now. 😉”
Apologies. It’s just that I’m trying to get you to try “beginner’s mind” to help you “empty your cup” and see/think more clearly about the problem from a different perspective. (So far, you seem more intent on defending your perspective than gaining a new one.)
When you say “branches isolate at the source-code” level, *what* exactly is the branch isolating? (Do you mean it is isolating “work” at the code-level?). I think you mean in the codebase (the VCS system).
That is a different structure/dimension than the code itself. Code structure (and behavior) is within the domain of refactoring. Branching takes place in the versioning structure/dimension. They may be versions of code, but you are creating/alternate conditional paths in something other than the code structure when you branch.
Toggles are quite literally conditional statements in the code itself. That is the code-level. The key to understanding is *when* is the corresponding “condition” getting evaluated (Binding time) and what is/isn’t being delayed as a result.
– version toggles/branches live in version-space, isolate development-paths *prior* to integration/merging, and are evaluated at checkout & checkin time (prior to even compilation).
– code toggles/branches live in code-space, isolate code-paths, and are evaluated execution-time (well after integration).
Sorry if that seems condescending, but it represents a different perspective from your current one, that can open-up your mind to seeing and thinking about the problem differently, and from a less extreme/hardline (and less limiting) mindset.
Feature-branches isolate feature *development* (not code) *before* the result is integrated/built; Feature-toggles isolate feature *execution* (in code) *after* the result is deployed/installed.
We at least agree the goal is to promote and enable more frequent (“continuous”) integration (and ultimately delivery) in shortest sustainable time (fast-flow) with the least amount of waste/complexity.
Toggles let you avoid/bypass additional version-branches and remove fear of delivering partially implemented/tested features (to production) to enable faster and more frequent integration-feedback. That comes at a cost of considerable added complexity to the code-structure, which adds more risk & difficulty to both merging changes to the affected code, but also of making *any* changes to the affected code. (And also require additional discipline to remove the toggles before too long, and/or additional tools to better manage code-toggles).
Branches let you avoid additional code-branches, but they don’t remove fear of delivering partially implemented/tested feature to production. If anything, they enable that fear (which increases the likelihood of delayed integration feedback).
Both still require discipline to integrate frequently (continuously). But *WHAT* is the added complexity & discipline to integrate continuously with the addition of feature-toggles versus with a feature-branch?
You seem to think one of them requires an extra manual step (to commit), for every integration. Even if it were true (albeit automated and not-manual), how does that compare to the added complexity of implementing feature toggles?
Each approach has a reasonable fear/risk associated with it of added complexity, delayed feedback, and of completing the effort before too much technical-debt (including integration debt) has piled up.
The most significant difference in the production result is whether the need to delay execution/availability of the feature is really and truly necessary!!! What is the real problem/fear/risk of not doing *either one* of feature-branch or feature-toggle and just integrate frequently.
Because if it really is truly necessary, then feature-toggle is *clearly* the way to go. But if it’s not, and if its just fear/uncertainty/doubt, then CI with FB is far less additional effort & complexity than CI with FT.
Brad, I think that I am open-minded, but when I have a strongly-held belief, it takes a strong argument to change it.
Someone saying “[FALSE]” is not enough for me 😉
You ask a good question, which is what I was trying to get to in my blog-post “When you say “branches isolate at the source-code” level, *what* exactly is the branch isolating? (Do you mean it is isolating “work” at the code-level?). I think you mean in the codebase (the VCS system).”
The things that you mention are all really only mechanisms. I tend to think about this stuff at a fairly abstract-level. The isolation that I am describing is fundamentally the isolation of ideas. Branching is, ultimately, designed to hide ideas from one-another while they are being worked on. More concretely, that takes the form of ideas in code – any code, source, config, binary-dependencies, anything.
I believe that CI is promulgated on the concept of minimising the duration of that hiding (isolation) of information. We want to continuously (or at least a reasonable approximation of “continuously”) test our ideas alongside everyone else’s. I think that we are agreed that that is a “good thing”.
Once we think that is a “good thing” then I believe that is a good idea to optimise for that. It is not the only thing, but in my experience if we optimise for speed and quality of feedback as a kind of “prime-directive” we get all sorts of other good things as side-benefits.
One example, you can’t really make these small, fine-grained, frequent, changes very easily if your code-base is a spaghetti-mess. Optimising for speed of change applies a pressure towards more modular code with better abstractions.
I believe that the speed, and quality, of feedback is a cornerstone of something that we may consider an “engineering” approach to software development. My experience, over many many projects and observing more, is that when people practice this fast-feedback the create better software faster.
So the speed and quality of feedback is *really* important. As a result I will work to maximise that. It is not the only thing that I do, but it is a guiding principle that is at the core of my approach to development. If I want to maximise the feedback, what I am looking to achieve is the fastest answer to my question “is my code ready to go live” that I can get.
If I want that answer fastest, I will attempt to remove any accidental complexity that gets in the way. I will grudgingly accept that some essentially complex things need to happen, I need to compile, I need to run tests, I need to delivery good feedback, but I will work really quite hard to make those things as efficient as I can. So, optimise the hell out of the essentially-complex things and eliminate the accidentally complex things.
I see branching, of any form, in two ways. If the branches are long-lived they compromise principle 1, they hide change and so are risky. If they are short-lived they may avoid this fate, but at that point they compromise principle 2, they slow things down unnecessarily for no advantage to me – they add accidental complexity and so I will eliminate them.
If I work on head, using git, my normal workflow, working on Trunk/”master” is going to be:
1. Make a change in my code.
2. git add *
3. git commit -m “some message”
4. git push
If I work on a branch…
1. git checkout -b “my_new_branch”
2. Make a change in my code.
3. git add *
4. git commit -m “some message”
5. git checkout master
6. git merge my_new_branch
7. git push
I am sure that there are 17 other ways to do this in git, there always is, I am sure that I will get suggestions to optimise this process – I took this from here: https://git-scm.com/book/en/v2/Git-Branching-Basic-Branching-and-Merging but my point is, whatever clever syntax you pick, there are simply more ideas involved in dealing with a branch. As well as the change to your code, you have to consider the branch, so any use is going to be a bit more complicated, and so take a bit more time and do all of that for no advantage to me in the way that I work.
Does that explain what I mean?
I don’t think that I am being closed-minded here. I have worked with branches for more of my career than not, as a result I treat they with caution because I have been bitten by them several times. Modern tools have made the mechanics of branching simple. I don’t buy the argument though that this is consequence-free. Branches, like everything else, come with trade-offs. On the whole my experience has been that teams that rely upon them go slower and get worse feedback than teams that don’t.
I am a believer in the value of “Beginner’s mind” and promote its use in the environments where I work. I don’t believe that what I am describing here is dogma, or a religious adherence to my preferred approach. Rather, I am trying to express my experience of trying several different approaches to this and finding one that, for me and the teams that I have worked with, works considerably better than the others. I have done the experiments here and rejected the practices that didn’t work out for me and the teams that I have worked with.
Hi Dave! (Sorry for the delay in response. I know you are at GoTo Chicago this week. I’ll be there Thursday; if youre there the same day maybe we can catch each other.
I think your basic definition of what a branch is (its purpose) is still missing something fundamental (and mischaractering something else that is also fundamental, yet separate). You keep equating isolation/segregation with *hiding* when it simply is not the case. I understand you mean *delaying* feedback from synchronization (integration), but this is completely different from hiding work/change. Hiding means it is not visible, and not accessible (not that it hasnt been integrated). If it is in a commit in the repository, it is visible, transparent, even accessible.
The *real* hiding is when the work is not visible or accessible in the central repository (either because it is in your workspace and/or local repo (in a DVCS)). This not only hides it’s content from the rest of the team, it hides its very existence (and even its intent). No one can see your in-progress changes, view them, review-them, or even tell the exist unless and until they exist in the central repository (where they dont have to be integrated in order to see them or access them).
Having them exist in the central repository, even when not synchronized yet, is what exposes the work. Makes it visible, viewable, accessible, and gives even more transparency by declaring its intent (as well as revealing its content), *and* its history, for the rest of the team to see.
In VCS terminology you will often see the terms “version”, “revision”, and “variant” where “version” is the more generic/abstract term, and the difference between the two types of versions is that:
– a “revision” is a version that is created with the intent of replacing its predecessor version when it is checked-in to the (central) repository
– a “variant” is a version that is created with the intent of coexisting along side its predecessor version when checked-in to the (central) repository.
Creating a branch declares a variant path (however temporary) that exists *concurrently* (at the same time, in the same place [content location]). There is no *hiding* here. It is intended to be visible, transparent, accessible, and all those other things that transparent (not hidden) information is supposed to be, and in fact is. That fact that it may not yet be ready to integrate to the main-trunk in the central-repository is not altered by the fact that it lives on a branch. The difference is that it lives *concurrently* in the same repository at the same time, whereas the same version fo content that exists only your workspace (or local repo) is *genuinely* hidden and it not allowed to (*visibly*) co-exist in the same repository at the same as a version that does exist (also not integrated) in the same repository.
If anything, the branch is the *opposite* of hiding. It publicly exposes/reveals the content in the central-repository while allowing it to co-exist at the same time+(file/code)-position on a variant development path.
There is no delayed integration-feedback as a result of this visible co-existence in the repository. When you are ready for it to be synchronized (integrated) to the main-trunk of the central repository it is no less difficult, nor any more steps to do so if it previously existed *only* in the workspace (and perceived additional “checkin” prior to commit is easily automated/removed by either the VCS-client [command-line or GUI] *or* the CI-tool (or both). You can codify it in an equivalent number of commands or operations either way.
The only difference is the *intent* (whether or not you *intended* to integrate frequently, or not until the end of the task/feature/story) and that intent is neither caused by *nor* determined by whether it exists on a branch. And even still *nothing* is *hidden*. It *might* be delayed (if that was the intent), and that fear of checkin/commit to the main-trunk of the central-repo has nothing to do with whether or not unintegrated content lives inside or outside of the central-repo.
Calling it “hiding” suggests a fundamental misunderstanding between *where* something is happening/visible versus *where* it is happening/visible. If its not visible in the central repo, *that* is hiding. If it *is* visible in the central-repo, its not hidden. It may be isolated, but its visible (unlike the content in the workspace or local repo). Delaying integration feedback is not the same thing as *hiding*, and the branch is not the cause either way. The branch operates at the *repository* level and its effects are visible (in the central repository) at that time (not hidden).
What you are calling code-level isnt a *level* (otherwise you should be able to tell me what is smaller/lower versus bigger/taller). It is a *time* (not a level), and specifically a *binding* time. The branch changes the timing of when the content is visible in the repo (not the timing of when it is integrated to the main-trunk). Its effects don’;t isolate anything at the *level* of the code, but at coding-*time*. (and its not hidden when it does it, but visible in the central repo for all the team to see).
Its more accurate to think of branches as a “bounded context” of *concurrency* in the repository (not of code/content in the repository). Whereas toggles are a “bounded context” of codestatements (and they can exists at the logical level, like classes, methods, packages; or at the code-level, like statements/lines, files, and folders).
If youre familiar with Neal Ford’s, Pat Kua’s, and Rebecca Parson’s work on “evolutionary architecture” (and I’m talking about since ~2015 or so, that explicitly includes the notion of “fitness functions” and “multiple dimensions” of concerns/views that can be subject to isolation/variation/change) then I suggest doing an experiment where you express different dimensions of concern for where+when branches operate, where+when merges operate, and where+when toggles operate. They have some overlap
– repositories include version dimension, including branches+tags+ commits; as well as filespace dimension of files & folders
– the code itself has both a logical dimension/view (methods, classes, packages, and different methods for code commonality & variability like conditionals, genericity, polymorphism) *and* physical (filesystem) dimension or files, folders, etc.
– then there is the time/evolution dimension itself, as it relates to the concurrency, parallelism, and distribution of changes and change-flow
– which is different from the dimension of concurrency, parallelism and distribution data and (executable) behavior of code (which is not only a different binding-time, but in a different stakeholder view).
I started characterizing these dimensions of “SCM” into a 4+2 “views” model back in the late 90s, and then compared them against “software architecture” views inn an old blog-entry at http://bradapp.blogspot.com/2006/12/dimensions-and-views-of-scm.html). I did this at the time because I felt the way we use and work with our SCM/ALM tools can have architectural impacts (like “technical debt”) and I wanted to be able to represent their complexity in other “development views” so I could visualize other kinds of patterns (specifically SCM patterns) and what the equivalent of “refactoring” and SOLID principles was for them.
Once you have those views/dimensions, try expressing the “fitness functions” that would tell you when/if branching is a better fit than toggling for each of the three different problems that a toggle, a development branch, and an integration branch are trying to solve (and at the right granularity of (micro)task, versus story, versus feature). This should not only reveal when+where toggling is more approriate than branching, but also at what granularity. For example, when, if ever, would you see yourself recommending “toggles” at the story-level (or smaller) as opposed to the feature-level.
Okay – last comment. This time on want to focus more on where are thinking is more alike than different (but more clearly delineate the “context” in which my thoughts apply, since I dont beleive “never branch” is incorrect (even tho I would still deprecate *feature* branching almost all the time, since I dont like either kind of feature branch much, even tho I recognize they solve different problems in different contexts of applicability).
I totally agree that branching is not without consequences. I’m often off the “everything’s a tradeoff” mindset, especially when it involves adding another level of indirection somehow). I’m also of the opinion that toggles are not without consequences too (and I don’t take those lightly either).
I’m not a fan of feature branching in general (not for feature *development*, nor for feature *integration*). The bigger issue is that each of the approaches is solving a *different* problem, neither one solves the problem the other solves (*when* it is done properly, and in the appropriate context).
– feature *integration* branches were designed to solve a problem using a “nested synchronization approach” in the context of multiple (feature) teams, and a sufficiently large repository to enable fast-frequent-feedback at the team-level (e.g., on the order of a few minutes to a few hours) while cross-team integration and *full* build+test happened anywhere from 1-4x per day). It had no place being used in a single (small-ish) team, and integration from feature-branch to main-trunk was still supposed to be happening at least daily. Pretty much all other uses of it were antipatterns/abuse-cases (still are).
Feature *development* branches started popping up (especially in a single-team scenario, where I have one developer [or pair] working on the feature, not a whole team) , and not only did “feature” refer to something much smaller, but someone got the silly idea that they shouldnt be integrating *at* *all* until the whole feature is complete.
In a small agile team, it can make sense to have “task-based-development branches” where the branch is used for the duration of a story (and always integrates still mid-story, after every red-green-refactor-commit cycle). Those do not suffer the ills you ascribe to them. But creating a separate branch for every commit-cycle (that lasts only hours, or even just 1 day) is overkill for the number of branches+refs it creates in the repo.
So called feature *toggles* solve a different problem. And that is when, for some *legitimate* reason it is not acceptable to deliver correctly-working but partially complete features (and I mean *features*, not stories) without the ability to *easily* “back them out” somehow *before* the feature implementation is complete (integrated+built+tested). More often than not, the reason is not genuine/legitimate, but simply fear, and very-frequent integration+test is all that is needed. HOWEVER, the more frequently you can deliver+deploy (without having to support+maintain legacy releases) the more it can make the need for decoupling feature integration from feature delivery/execution be legitimately desirable.
Feature-Toggles let you integrate *continuously*, even deliver+deploy continuously, but allow delaying release (of *executable behavior*) to be all-or-nothing (rather than incremental) *without* forcing integration to be all-or-nothing later (rather than incremental). Toggles *decouple* code-integration from behavior-execution (the later happens at later binding-time of code, not a different “level” of code).
> If “both”? you are on a journey into exponential complexity growth as you add more flags – Flag “A” on Flag “B” on, Flag “A” on Flag “B” off, and so on! This is a never-ending game that you can’t win in any definitive way.
My current company has an interesting approach to this where there is only 1 feature flag called “incubating features”. You either turn it on (and are exposed to everything the is incubating) or off (and have the experience live users do).
When I joined them was quite surprised since I had always only worked with separate flags per feature, like you describe above.
Its working quite well for us nas, in practice, I’m finding it much simpler and easier to work with than separate flags:
• No combinatory effect
• Immediate integration of all features in progress and very early detection of clashes
• Easy for everyone to see and use what is in progress: it’s just one well known flag
• Impossible to forget to take functionality from behind the feature flag — that’s the only way to release it to all users (turning on a ff in live and not doing the code removal *was* a pattern I’ve fought against many times in the past in other places)
• We use code level annotations to mark each usage of the feature flag in the code with what functionality it belongs to, so we can easily leverage the compiler when it’s time to remove it
I love it! A great example of Keep It Simple Stupid!
This sounds functionally very similar to having a separate “stage” or “test” environment that is always running the latest in-development code but without all the WET redundancy of running multiple environments.