When Jez and I wrote our book, we knew that we were describing a powerful approach. We were very nervous of claiming a “Methodology” though. Instead we saw the Continuous Delivery book as describing, in some detail, an approach to Build, Test and Deployment Automation and hinting at something broader in-scope.
I am less reticent these days. I have seen the philosophy of Continuous Delivery transform organisations. I have been personally involved in helping many firms make this shift. I now make no bones, CD is a holistic approach that extends a long way beyond merely Build, Test and Deployment Automation.
Working so that you are in a position to deliver value into the hands of your users and customers continuously is a radical change, but it is now measurably, demonstrably the most effective way that we humans currently know how to create great software.
The impact on teams and organisations is profound. Continuous Delivery as a discipline is not just about the technicalities either in terms of practice or impact.
From my own experience, but also backed-up by over 10 years of robust research, studies and the experience of software developers around the world, I now know that Continuous Delivery has the following benefits:
- We can create better software faster, with no trade-offs between those two ideas.
- We can reduce defect counts by multiple orders of magnitude.
- We can spend a significantly greater proportion of our time on new ideas and less on unplanned tasks.
- We can innovate more quickly and steer our businesses towards greater financial success.
- We can solve harder problems.
- We can have a better work/life balance while doing all of these things.
- We can work with less stress, and more creativity.
- We can be more compliant and safer in regulated and safety-critical industries.
- We can revolutionise the organisations in which we work.
- We can do all of these things whatever the nature of the software.
I now believe that this engineering-led approach is not only generally applicable, but is the best way to create any software.
I asked some people that I know, who have personal experience of employing Continuous Delivery in businesses of all sizes, to comment on this impact:
“Continuous Delivery helps us focus effort on things that bring value to our customers, instead of wasting time on repeatable and automate-able tasks in configuration, testing and deployment. It enables us to move fast with confidence, frequently applying small changes and keeping our product quality high and deployment risk low. MindMup.com is a product of a two-person team, serving millions of users across the world. The two of us do everything from user research through development, testing and operations tasks, to customer support. There’s no way we’d be able to achieve any kind of delivery speed if our releases required a lot of work or caused customer support requests. Software quality and stability are essential so we can focus on adding value instead of fixing bugs. Investing in continuous delivery practices pays off big because it delegates repeatable tasks to machines and frees up our time to work on things that actually require human insight, allowing us to successfully compete with several orders of magnitude larger organisations.”
Gojko Adzic (http://gojko.net)
The automation that Gojko describes is a central practice in CD. I like that he uses the word “Investing” though. I speak to many teams starting out who claim that they don’t have time to automate. This is a bit like saying that you are going to walk from London to Edinburgh because you don’t have the time to put petrol in your car.
The State of DevOps reports says that teams that practice CD spend 44% more of their time on new work than teams that don’t.
“Working with large enterprises, I’ve seen that the technical agility Continuous Delivery supports can be leveraged for greater business agility. With that in mind, it was easy to believe Dr. Forsgren’s research showing better business performance from teams adopting DevOps. I put my money where my mouth is, so to speak, and invested in companies who convince me that they’re dedicated to DevOps and Continuous Delivery. My only regret to date is not investing more.”
Eric Minick (IBM) (https://devops.com/revisiting-my-bet-on-devops/)
Since the publication of my book, I have grown to believe that the reason that CD works is quite profound. I believe that CD is founded on the application of scientific principles to solving problems in software. We make evidence-based decisions, use falsification, form hypotheses that we test in the form of (often automated) experiments. We proceed in small steps, validating our learning and understanding as we progress. We control the variables with techniques like Infrastructure as Code.
For me CD is a genuine “Engineering Discipline for Software“. I too find Dr Fosgren’s work (described in the excellent “Accelerate” book) compelling. We have a measuring stick (Stability & Throughput) and correlative model that can guide our efforts to learn to create “Better Software Faster“.
“Working on a software product with long upgrade cycles, my team felt caught between sales demanding features immediately and customers who couldn’t immediately adopt and didn’t want frequent releases. We rearchitected to enable more of a Continuous Delivery approach and zero downtime updates to just one area of the product that was a magnet for these sales requests. That investment gave us the agility we needed where we needed it. The developer/sales relationship improved as did the business.”
Eric Minick (IBM) (https://dzone.com/articles/know-the-business-case-for-rearchitecting)
Eric points out that this is NOT just a technical impact. CD has a revolutionary impact on the way in which the business that employ it can operate.
Organisations that practise Continuous Delivery “have a 50% higher market cap growth over 3 years“
“I would love to say that it made NS better, faster, cheaper, and happier – although many believe this we still can’t really prove that with data! I can say however that our CD efforts in 2016/17, and the Agile and DevOps transformation that followed, has led to greater ownership by the teams, to fundamental discussions about organization and governance of software development, and to improved understanding and alignment between business and IT. And of course, in the meanwhile, teams have automated everything they could, supported by a fine tool suite and plenty of coaches.”
Huub van der Wouden
Dutch National Railways
The adoption of Continuous Delivery is not easy. Because of its holistic nature the changes sometimes proceed slowly, but even then the impact is significant. The data from CD teams is interesting in terms of cultural performance as well as technical. Teams that practise CD claim better work/life balance, lower stress, and a greater sense of ownership.
The State of DevOps Report found that “The No.1 predictor of high performance in teams is Job Satisfaction“.
“At Siemens Healthineers, we have a growing number of teams adopting Continuous Delivery in the heavily regulated medical device industry. Currently, we have 20+ teams on the journey to Continuous Delivery. Different teams adopt various Continuous Delivery ways of working at different rates. Some teams are further ahead than others. What we have seen over the years is a growing appetite for Continuous Delivery, which is set to continue in future. Teams making breakthroughs in e.g. testing, deployment etc. never look back but rather want to achieve more!”
Vladyslav Ukis
Siemens Healthcare
Vlad describes another, growing, impact of CD. It is almost impossible to imagine the creation of a Deployment Pipeline without getting a perfect audit-trail of production changes as a side-effect. This, and other properties, make CD the perfect approach for regulated and safety-critical industries. I wrote about this in this blog post on “Continuous Compliance“.
“On the cost side, the software delivery has to be optimized for small chunks, in order to support the revenue side of the story. This requires the delivery organization to be set up in such a way that development teams can make releases independently. This holds true irrespective of the number of teams. That is, the cost of running the organization (overhead) does not significantly increase when the number of teams increases. Additionally, the teams can be trained to work in a way that allows them to accelerate over time. This reduces the cost per small batch delivery over time.
On the capital allocation side, the software delivery in small chunks has another business advantage. If an organization has a software delivery capability with small chunks, it enables investments in software products to be placed with a stop option, which can be exercised at any time. This way, you can invest a little, see whether the results are promising, invest a little more if they are, reduce the investment if they are not, and keep going this way. It is an efficient way to allocate the capital bit by bit taking small risks and evaluating results along the way. The stop option allows the capital to be invested with an in-built risk reduction strategy.
For the reasons above, I would not want to invest in software products that are not built using Continuous Delivery!”
Vladyslav Ukis
Siemens Healthcare
Working in smaller steps is natural in CD. It gives faster, clearer feedback of each change and is promoted by an efficient Deployment Pipeline. It is is also extremely valuable to businesses. It allows them to try new ideas at lower cost, to be more reactive to customer or market demand. It also allows them to take the safety of the systems that they create more seriously too. If a change is small and simple, it is also lower risk, easier to diagnose if something goes wrong and easier to remove if necessary.
In a paper on “Online Experiments at Large Scale” looking at Microsoft – “2/3 of ideas produce zero or negative value“
“CI is a communication tool. It alerts people working on a system that they need to talk to other people to resolve a potential conflict between the work they are doing separately. Or at least, appeared to be doing separately until CI detected the work wasn’t as separable as they thought. That’s why integrating “at least once a day” is not a useful definition any more. Do you want to waste a day’s work before dealing with any conflicts? CI should provide that alert as fast as possible.”
“Integration should happen so frequently that when a conflict occurs, it’s quicker for the people involved to have a discussion, discard some of the changes and write them again than it is to merge their changes.”
“Continuous Deployment changes the way you can think about architecture. It gives you more choices about where data and logic can be placed. When you can safely/rapidly/automatically redeploy any components of an application at any time, you can choose to place data or logic in those components rather than in separate services or databases.”
“CI/CD pipelines are not a “dev” environment. They are critical to the functioning of the production system, because they are the way fixes get deployed into production when there is an incident.”
Nat Pryce (http://www.natpryce.com)
Nat points out the profound impact that thinking about fast-feedback and test-ability has on the software that we create. In general “quality” in code and systems has been an issue of experience and talent. Continuous Integration, and its big brother Continuous Delivery, apply a separate pressure for higher quality.
If you can’t release into production if a single test fails (a recommended CD practice) then keeping the tests passing is central to the approach. To keep the tests passing you need “repeatable reliable” tests. If you want your tests to be deterministic, then you need to make your code testable. The properties of testable code are the properties that we value as the hallmarks of high-quality software.
Of course this still depends on the talent and ingenuity of development teams, but it applies a pressure to do the right things that otherwise only exists as a matter of personal discipline. CD is a GREAT tool to drive a better focus on higher-quality work and provides a mechanism to give clearer feedback to developers on the quality of their work. (I described this idea, in part, here)
“I highly recommend this book <Continuous Delivery>. Working at LMAX where we put into practice many of these things fundamentally changed the way I think about software development”
“I’ve seen that organisations that have adopted CD spend significantly less time on releases. Higher levels of automation means much less (if not zero) time spent by developers and operations team members on evenings and weekends. This is beneficial in a number of ways: less overtime; less time spent on releases means more time to develop features; less human interaction means a lower risk of human-introduced errors; and a happier, more productive team environment (which is great for recruitment and retention).”
Trisha Gee, JetBrains
Trish describes how the technical practices of CD reinforce the desirable human outcomes for the people working on these teams. Less stress, better work/life balance and generally a better, more creative, higher-quality work environment.
“Our CI/CD environment is consistently humbling our development team with failing tests that were not expected. Most of these failures would have silently made their way to production unnoticed.”
“Our CI/CD environment is able to provide fast accurate feedback because our developers are empowered to write tests that stand up to a rigorous cost benefit analysis rather than simply adding tests to meet some arbitrary code coverage metric”
Judd Gaddie (TransFICC)
Judd’s team operates a sophisticated CD approach delivering very high-class software. CD is an engineering discipline in that if you follow the practices your will certainly get a higher-quality result. Measured in terms of speed and quality.
“We are able to make large, cross cutting changes to our system with high-levels of confidence due to the quality of our CD pipeline. We were able to completely re-write the architecture of our system and know it worked from our pipeline (we didn’t have to change any acceptance tests either!).
Some things that spring to mind but are not necessarily CD related…The first thing we built when we started the company was our CD pipeline and a feedback screen. Whilst it meant we sent slowly at the beginning it has meant we have always had a strong testing pipeline and allowed us to focus on automating everything and provided an easy way of showing whether our software works or not. It has also provided credibility to the company in a way of showing we know what we are doing
Tom McKee (TransFICC)
Zero time spent “preparing” a release as trunk is always releasable”
Tom takes us back to the start with the idea of investing in our approach and our work. Work done to make us more efficient is not waste, it helps us to eliminate waste. For Continuous Delivery teams there is no trade-off between speed and quality. In fact investing so that you can go quickly naturally leads to working in smaller steps and promotes higher-quality outcomes.
I believe that Continuous Delivery represents the “state of the art” for software development. I also believe that this approach is a genuine “engineering discipline” for software development.
Continuous Delivery does what engineering does in other fields, it amplifies our craft and creativity and allows us to create machines and systems with greater speed, quality and efficiency. There is no better way, that we know of, to create high quality software efficiently.
It takes hard work, and often ingenuity, to adopt Continuous Delivery. To be world-class you need to consider cultural, organisational and technical performance. Organisations that are world class at this are different in approach, and effectiveness, to organisations that aren’t. This is the approach behind many of the leading companies in the world.
So, ten years after my book was first published, I am very proud of the impact of these ideas, and in my part in popularising them, but there is still a lot more work to do, so I am looking forward to the next 10.
You can find out more about some of these ideas on my “Continuous Delivery” YouTube channel here.
I have also recently released an on-line version of some of my successful CD training courses here:
Finally, if you haven’t had enough of me yet, you can subscribe to my mail list here.