I recently attended the Devoxx conference. One of the speakers was talking on a topic close to my heart, Continuous Delivery. His presentation was essentially a tools demonstration, but one of the significant themes of his presentation was the use of feature-branching as a means of achieving CD. He said that the use of feature-branching was a debatable point within the sphere of CD and CI, we’ll I’d like to join the debate.
In this speaker’s presentation he demonstrated the use of an “integration branch” on which builds were continuously built and tested. First I’d like to say that I am not an opponent of distributed version control systems (DVCS), but there are some ways in which you can use them that compromise continuous integration.
So here is a diagram of what I understood the speaker to be describing, with one proviso, I am not certain at which point the speaker was recommending branching the “integration branch” from “head”.
In this digram there are four branches of the code. Head, the Integration branch and two feature branches. The speaker made the important point that the whole point of the the integration branch is to maintain continuous integration, so although feature branches 1 and 2 are maintained as separate branches, he recommended frequent merges back to the Integration branch. Without this any notion of CI is impossible.
So the Integration branch is a common, consistent representation of all changes. This is great, as long as each of these merges happens with a frequency of more than once per day this precisely matches my mental model of what CI is all about. In addition, providing that all of the subsequent deployment pipeline stages are also run against each change in the integration branch and releases are made from that branch this matches my definition of a Continuous Delivery style deployment pipeline too. The first problem is that if all of these criteria are met, then the head branch is redundant – the integration branch is the real head, so why bother with head at all? Actually I keep the integration branch and call it head!
There is another interpretation of this that depends on when the integration branch is merged to head, and this is what I think the speaker intended. Let’s assume that the idea here is to allow the decision of which features can be merged into the production release, from head, late in the process. In this case the integration branch, still running CI on the basis of fine-grained commits, is evaluating a common shared picture of all changes on all branches. The problem is that if a selection is made at the point at which integration is merged back to head then head is not what was evaluated, so either you would need to re-run every single test against the new ‘truth’ on head or take the risk that your changes will be safe (with no guarantees at all).
If you run the tests and they fail, what now? You have broken the feedback cycle of CI and may be seeing problems that were introduced at any point in the life of the branches and so may be very complex to diagnose or fix. This is the very problem that CI was designed to eliminate.
Through the virtues of CI on the integration branch, at every successful merge into that branch, you will know that features represented by feature branches 1 and 2 work successfully together. What you can’t know for certain is that either of them will work in isolation – you haven’t tested that case. So if you decide to merge only one of them back to head, you are about to release a previously untested scenario. Depending on your project, and your the nature of your specific changes, you may get away with this, but that is just luck. This is a risk that genuine CI and CD can eliminate, so why not do that instead and reduce the need to depend on luck?
Further, as I see it the whole and only point of branching is to isolate changes between branches, this is the polar opposite of the intent of CI, which depends upon evaluating every change, as frequently as practical, against the shared common picture of what ‘current’ means in the system as a whole. So if the feature branches are consistently merging with the integration branch, or any other shared picture of the current state of the system – like head, then it isn’t really a “feature branch” since it isn’t isolated and separate.
Let’s examine an alternative interpretation, that in this case I am certain that the speaker at the conference didn’t intend. The alternative is that the feature branches are real branches. This means that they are kept isolated, so that people working on them can concentrate on those changes and only those changes without worrying about what is going on elsewhere. This picture represents that case – just to be clear, this is a terrible idea if you mean to benefit from CI!
In this case feature branch 1 is not merged with the integration branch, or any other shared picture, until the feature is complete. The problem is that when feature branch 2 is merged it had no view of what was happening on feature branch 1 and so the merge problem it faces could be nothing at all or represent days or even weeks of effort. There is no way to tell. The people working independently on these branches cannot possibly predict the impact of the work elsewhere because they have no view of it. This is entirely unrelated to the quality of merge tools, the merge problems can be entirely functional, nothing to do with the syntactic content of the programming language constructs. No merge tool can predict that the features that I write and features that you write will work nicely together, and if we are working in isolation we won’t discover that they don’t until we come to the point of merge and discover that we have evolved fundamentally different, incompatible, interpretations. This horrible anti-pattern is what CI was invented to fix. For those of us that lived through projects that suffered, all to common, periods of merge-hell before we adopted CI never want to go back to it.
So I am left with two conclusions. One, for me the definition of CI is that you must have a single shared picture of the state of the system and every change is evaluated against that single shared picture. The corollary of this is that there is no point having a separate integration branch, rather release from head. My second conclusion is that either these things aren’t feature branches and so CI (and CD) can succeed, or they are feature branches and CI is impossible.
One more thought, feature-branching is a term that is, these days, closely associated with DVCS systems and their use, but I think it is the wrong term. For the reasons that I have outlined above these are not real branches, or they are incompatible with CI (one or the other). The only use I can see for a badly mis-named idea of “feature branching” is that if you maintain a separate branch in you DVCS, but compromise the isolation of that branch to facilitate CI, then you do have an association between all of the commits that represent the set of changes that are associated with particular feature. Not something that I can see an immense amount of value in to be honest, but I can imagine that it may be interesting occasionally. If that is the real value then I think it would benefit from a different name. This is much less like a branch and more like a change-set or more accurately in configuration management terms a collection of change-sets.
DO branch-per-feature.. properly. Not what you represented here.
You set yourself up for disappointment when you want to release and can’t take out a feature. We integrate as often as any other team. It comes down to discipline. Branch-per-feature is an awesome way to work.
Our process is a bit different. First of all, we don’t release from the integration branch. The idea behind branch-per-feature is that you pick the features that are ready and passed everything on an integration branch (includes QA). You don’t merge anything from integration branch to master/head or realease or whatever you choose to call it.
Organizing your work is a good thing. Things like OCP and other S.O.L.I.D. principles make this easy. If you are having a hard time with branch-per-feature, it’s probably because all you can handle is trunk-based dev.
Here is more detail about it:
Adam, thanks for the feedback, even if we do disagree 😉
I read through the discussion on Google+ but I struggle to see what benefits you get for the complexity of keeping features isolated. To be clear I am not talking about the complexity of the tool use, but the complexity in terms of the number of different versions in play. Even if the tools are wonderful, and I know that DVCS are very strong in this area, there is an overhead in keeping a coherent picture of what branch is in what state.
It seems to me that you must either evaluate a release change set very late in the process and so are taking a risk that you may have a problem, or your isolated branches offer no benefit – what am I missing?
Also, the benefit to keeping branches for a feature is that you can make an ad hoc build at anytime. This is not the case with trunk based. It also organizes your work. It’s clear what was done to implement a feature. You’re not filtering based on commit messages. You could turnfeatures on or off with feature toggles, but I find them the biggest hack of what’s in ci these days and a potential source of errors that you simply could not have if you do branch per feature properly.
What you are missing? You are missing that if you want to have longrunning feature branches (which is a practice I would advise against), then you must merge your integration branch to your feature branch and run the tests on your feature branch. You don’t merge and hope that nothing will conflict – you pull in the changes from the parent branch as often as possible just to be certain there are no conflicts.
Both of your pictures totally miss this flow of changes outward. Noone disagrees that long running branches that isn’t up to synch with changes in parent branches are bad – so don’t do that. You pay some lipservice to DVCS’s, but you don’t seem to understand how the changes in your workflow that a VCS that can branch without being hard and error prone enable you to do shortlived featurebranches.
Sorry that came out a bit harsher than I intended. I apologize for that, but I stand by my opionion that you seem to argue against a way of working that noone else suggests anyone should use.
My pictures show what was described by the speaker, not my idea! My point is that if you merge back frequently (more than once per day) they aren’t branches in any meaningful sense. They are not isolated. I agree that there can be a benefit of tagging a set of related changes, but that isn’t a branch.
There are cases where two people working independently can come up with features that don’t work together. This is irrespective of the technology. If you and I both deliver the same function in two different ways that, say decrements a counter in response to the same event, then we have a functional clash we get a double decrement when we should get one. The code is fine, so unless your DVCS is a LOT smarter than mine, it can’t possibly detect such a clash. This silly example may point to other failures in the development process (people not talking to one another enough perhaps?, poor test coverage) but that is irrelevant. This is a type of failure that no merge tool can detect, but that a process of continuous integration can.
I think feature branching becomes alluring to some because it has the potential to delay the pain of working with potentially conflicting features and timelines until you decide to merge and integrate. Working from a single code-line requires careful thought and control but brings that pain forward surfaces it earlier .
This is precisely my concern, I don’t want to delay the pain of merging because it is, at least as, likely to get worse rather than better. I would prefer to take the pain of a merge now and so know where I stand. Bringing the pain forward seems much teh safer course to me.
The proper way of doing CI is to continuously merge. This is not against proper branch per feature though. I’m not happy that organizing your work gets lumped together with an obvious anti-pattern.
It’s not left till the end of the process. The QA branch gets remade with new features as they get completed. This build is always ready for deployment to production. I say remade because in the process you may reject another feature that previously was thought to have been completed. But most of the time, it can be just a merge. This branch has different parameters on the ci server. It produces deployable artifacts. The integration branch, which is used almost as often as one commits, is used for the feedback needed to see if everything is getting along. More importantly, it’s recording resolutions which can be used by other developers when running into the same conflicts or when ultimately merging into the QA branch. This branch should really be called an RC branch.
Hi Adam. I think the problem is most developers do NOT use branches the way you suggest. This is akin to arming civilians with firearms without proper training or licensing: chaos unfolds.
Due to the high risk of abuse of branches for most teams I agree with Dave: if you can’t use them carefully just don’t use them at all.
If you have a team that uses them, and you can get a release cadence your business is satisfied with then don’t fix what isn’t broken. Most teams have a broken process. We have teams that release changes to the business in months or even years where the business demands shorter turn around time. Clearly feature branches are misused here and the best strategy is to just not use them in those cases.
I find Trunk Based Development with Feature toggles are simpler for a team to use and implement than feature branches (correctly executed usage of them anyway). If you don’t have problems then keep using the feature branches. If you do have problems, try TBD.’
To my mid a rapid development strategy that includes feature branches is VERY difficult to execute. If your team can do that, my hat is off to you and all due respect extended.
For the rest of the industry, just use TBD with feature toggles and leave feature branches to those who need them and have the training and experience to use them wisely.