Best branching strategy when doing continuous integration?

What is the best branching strategy to use when you want to do continuous integration?

1. Release Branching: develop on trunk, keep a branch for each release.
2. Feature Branching: develop each feature in a separate branch, only merge once stable.

Does it make sense to use both of these strategies together? As in, you branch for each release but you also branch for large features? Does one of these strategies mesh better with continuous integration? Would using continuous integration even make sense when using an unstable trunk?

I find the topic really interesting since I heavily rely on branches on my daily job.

• I remember Mark Shuttleworth proposing a model about keeping the main branch pristine while going beyond conventional CI. I posted about it here.
• Since I'm familiar with Cruise Control, I also blogged about task branches and CI here. It's an step by step tutorial explaning how to do it with Plastic SCM.
• Finally, I found some of the topics about CI (and potentially talking about branching) at Duvall's book on CI very interesting too.

Hope you find the links interesting.

The answer depends on the size of your team and quality of your source control and the ability to merge correctly complex change sets. For example in full branch source control like CVS or SVN merging can be difficult and you might be better off with the first model, while if using more complex system like IBM ClearCase and with a larger size of team you could be better of with the second model or a combination of the two.

I personally would separate the feature branch model, where each major feature is developed on a separate branch, with task sub-branches for each change done by individual developer. As features stabilize they get merged to trunk, which you keep reasonably stable and passing all regression tests at all times. As you near the end of your release cycle and all feature branches merge, you stabilize and branch of a release system branch on which you only do stability bug fixes and necessary backports, while the trunk is used for development of the next release and you again branch off for new feature branches. And so on.

This way trunk contains always the latest code, but you manage to keep it reasonably stable, creating stable labels (tags) on major changes and feature merges, the feature branches are fast paced development with continuous integration and individual task sub-branches can be often refreshed from the feature branch to keep everyone working on the same feature in sync, while simultaneously not affecting other teams working on different features.

At the same time you have through the history a set of release branches, where you can provide backports, support and bugfixes for your customers who for whatever reason stay on previous versions of your product or even just latest released version. As with the trunk, you do not setup continuous integration on the release branches, they are carefully integrated upon passing all regression tests and other release quality control.

If for some reason two features are co-dependent and need changes done by each other, you can consider to either develop both on the same feature branch or to require the features to regularly merge stable parts of the code to trunk and then refresh changes from trunk to exchange code between trunk branches. Or if you need to isolate those two features from others, you can create a common branch off which you branch those feature branches and which you can use to exchange code between the features.

The above model does not make much sense with teams under 50 developers and source control system without sparse branches and proper merging capability like CVS or SVN, which would just make this whole model a nightmare to setup, manage and integrate.

I personally find it much cleaner to have a stable trunk and do feature branching. That way, testers and the like get to stay on a single "version" and update from trunk to test any feature that is code complete.

Also if multiple developers are working on different features, they can all have their own separate branches, then merge to trunk when they're done and send a feature to be tested without the tester having to switch to multiple branches to test different features.

As an added bonus, there is some level of integration testing that comes automatically.

Release branches are very useful, and even absolutely required, if you need to maintain several versions of your app.

Feature branches also are very convenient, notably if one developer needs to work on a huge change, while others still release new versions.

So to me using both mechanisms is a very good strategy.

Interesting link from the Book of SVN.

I think either strategy can be used with continuous development provided you remember one of the key principles that each developer commits to trunk/mainline every day.

http://martinfowler.com/articles/continuousIntegration.html#EveryoneCommitsToTheMainlineEveryDay

EDIT

I've been doing some reading of this book on CI and the authors make suggest that branching by release is their preferred branching strategy. I have to agree. Branching by feature makes no sense to me when using CI.

I'll try and explain why I'm thinking this way. Say three developers each take a branch to work on a feature. Each feature will take several days or weeks to finish. To ensure the team is continuously integrating they must commit to the main branch at least once a day. As soon as they start doing this they lose the benefit of creating a feature branch. Their changes are no longer separate from all the other developer's changes. That being the case, why bother to create feature branches in the first place?

Using branching by release requires much less merging between branches (always a good thing), ensures that all changes get integrated ASAP and (if done correctly) ensures your code base in always ready to release. The down side to branching by release is that you have to be considerably more careful with changes. E.g. Large refactoring must be done incrementally and if you've already integrated a new feature which you don't want in the next release then it must be hidden using some kind of feature toggling mechanism.

ANOTHER EDIT

There is more than one opinion on this subject. Here is a blog post which is pro feature branching with CI

http://jamesmckay.net/2011/07/why-does-martin-fowler-not-understand-feature-branches/

I've recently come to like this model when using git. Although your question is tagged "svn", you might still be able to make some use of it.

Continuous Integration can to some extent happen in the "develop" branch (or whatever you call it) in this model, though having long running feature branches for future releases wouldn't make it so rigid as to consider every change happening to code somewhere. The question remains, whether you'd really want that. Martin Fowler does.

As long as you understand principles, you can always re-invent the best practices. If you don't understand principles, the best practices will take you that far before falling apart due to some conflicting external requirement.

For best intro into the Mainline Model, read this: https://web.archive.org/web/20120304070315/http://oreilly.com/catalog/practicalperforce/chapter/ch07.pdf

Read the link. Once you got the basics, read the following article by venerable Henrik Kniberg. It will help you relate Mainline Model with continuous integration.

http://www.infoq.com/articles/agile-version-control

Continuous integration should not be any kind of a factor in determining your branching strategy. Your branching approach should be selected based on your team, the system under development and the tools available to you.

Having said that ...

• there's no reason why CI couldn't be used in both of the approaches you describe
• those approaches work quite well in combination
• neither of the two work "better" than the other
• CI makes total sense with an unstable trunk

All of this was answered in the fourth question on the page that you took the diagrams from: http://blogs.collab.net/subversion/2007/11/branching-strat/

When we started our team we inherited a release-based strategy from the vendor that originally developed the system we were about to get in charge of. It worked up until the time when our customers requested that several developed features should not be included in a release (f.y.i. ~250k lines of code, ~2500 files, Scrum with XP SDLC).

Then we started looking at feature-based branches. This also worked for a while - like 2 months until the point we realized that our regression testing process would take over 2 weeks which combined with the uncertainty of what would be released created a huge inconvenience.

The final "nail in the coffin" of pure SC strategies came when we decided that we should have 1. stable trunk and 2. Production should contain ST, UAT, and Regression tested BINARIES (not just source - think CC.)

This lead us to devise a strategy that is a hybrid between feature and release-based SC strategies.

So we have a trunk. Every sprint we branch out the sprint branch (for the non-agile folks - a sprint is just a time-boxed development effort with variable output based on complexity.) From the sprint branch we create the feature branches and parallel development starts in them. Once features are complete and system tested, and we receive intent to deploy them, they are merged to the sprint branch - some may float across several sprints, usually the more complex ones. Once the sprint is near its end and the features are complete ... we "rename" the sprint branch to "regression" (this allows CruiseControl to pick it up without any reconfiguration) and then regression/integration testing begins on the cc-built EAR. When that is all done, it goes in production.

In short, feature-based branches are used to develop, system test and UAT functionality. The sprint branch (really the release branch) is used to selectively merge features on-demand and integration-test.

Now here is a question to the community - we are obviously having trouble performing continuous integration because of the fact that development happens on many branches and the reconfiguration overhead of CruiseControl. Can someone suggest and advice?

Dave Farley, an author of Continuous Delivery, referred to Trunk Based Development (TBD) as the cornerstone of Continuous Integration (CI) and Continuous Delivery (CD). He says:

> Any form of branching is antithetical to Continuous Integration.

He also says,

> Feature Branching is very nice from the perspective of an individual developer but sub-optimal from the perspective of a team. We would all like to be able to ignore what everyone else is doing and get on with our work. Unfortunately, code isn’t like that. Even in very well-factored code-bases with beautiful separation-of-concerns and wonderfully loosely-coupled components, some changes affect other parts of the system.

Trunk Based Development (TBD) is the practice of integrating code changes into the trunk (a.k.a, master, mainline) at least once per day - preferably multiple times per day. Continuous Integration (CI) is a similar practice except that it also involves verifying the code changes using automated tests. The best branching strategy for this is to work directly off the trunk and to perform code reviews through Pair-Programming. If for some reason you can't pair, or you just really want to branch, make sure your branches are short-lived (less than a day).

> I work on Trunk, “master” in my GIT repos. I commit to master locally and push immediately, when I am networked, to my central master repo where CI runs. That’s it!

For large features (i.e. ones that take longer than a day), try to break them into small chunks of logic which can be integrated into the trunk without breaking the software. You can also use techniques such as feature-flagging and branching by abstraction which allow you to deploy incomplete work without affecting end users.

> I use branch by abstraction, dark-releasing and sometimes feature-flags. What I get in return is fast, definitive (at least to the quality of my testing) feedback.

The way I see it you want to have a limited set of branches where you can focus. Since you want tests, code quality metrics, and many interesting things to run with the builds, having too many reports will probably get you to miss info.

When and what to branch, usually depends on the size of the team and the size of the features being developed. I don't think there is a golden rule. Make sure you use an strategy where you can get feedback early/often, and that includes having quality involved from the very beginning of the features. The quality bit, means that as you are automating as the team develops, if you branch for a large feature set a team is building, you gotta have quality involved in the team as well.

ps Where did you get those approach references? - doesn't feel that those graphs represent all the options

Update 1: Expanding on why I said it isn't a golden rule. Basically for relatively small teams I have found it best using an approach that is a mix. Feature branches are created if it is something long and part of the team will continue adding smaller features.

I think the tools you use are a big factor here.

• If you are using subversion, sticking with option 1 and release from branches.
• If you are using GIT, option 2 will work well for you.