Science and Software Development

I have been talking about Continuous Delivery being, informally, an application of the scientific method to software development for several years now.

I have spoken about it, CD, being a candidate for the beginnings of a genuine engineering discipline for software development.

My interest in this is related to my interest, as an amateur, in science in general and physics in particular. I am an avid reader of popular science, but I am not very academically qualified in these subjects.

Nevertheless I think that there is something important, significant, here.

My interests have led me to read more deeply into some of these topics, I am learning more.

The Beginning of Infinity

Two things that have come together recently and made me want to write this piece, which has been brewing in the back of my mind for some time.

The first is that I was given a gift, a book that is probably the most mind-expanding book that I have ever read.

“The Beginning of Infinity” by David Deutsch is a profoundly deep work on the philosophy of science (and rationality). People are starting to talk of this book, this thinking, as the successor to the work of Karl Popper who’s ideas, in the 1930s, revolutionised the way that science has been viewed and practiced ever since. Popper was the person who described, amongst other things, the importance of being able to falsify theories.

The classic example from Popper is – we can never prove that all swans are white, but as soon as we see a single black swan we can disprove, falsify, the white swan assertion. These days a scientific theory is not really valid unless it is capable of being falsified.

There are too many ideas in Deutsch’s “The beginning of infinity” for me to summarise them all here, go and read the book – you can thank me for the recommendation later 😉 One of the key points though is that science proceeds by trying to establish what Professor Deutsch calls “Good Explanations”. A “good explanation” is an explanation that is hard to vary without changing its meaning and one that is falsifiable.

“There is only one way of thinking that is capable of making progress, or of surviving in the long run, and that is the way of seeking good explanations through creativity and criticism.”

“Its (science’s) quest for good explanations corrects the errors, allows for the biases and misleading perspectives, and fills in the gaps.”

“So we seek explanations that remain robust when we test them against those flickers and shadows, and against each other, and against criteria of logic and reasonableness and everything else we can think of. And when we can change them no more, we have understood some objective truth. And, as if that were not enough, what we understand we then control. It is like magic, only real. We are like gods!”

David Deutsch,
    “The Beginning of Infinity: Explanations That Transform the World”

Software Development, Science & Engineering

I think that this philosophy of science stuff has profound impacts on how we should approach software development and even how we view what software development is.

The second thing that made start on writing about this, was based on a passing comment that I made on Twitter. I repeated a viewpoint that I have long held that automated testing in software is best thought-of, used, as a falsification mechanism. Amongst several others Bill Caputo replied and included some links to his thoughts on this which very closely aligned with mine and described some of these ideas better than I had.

Then in the twitter conversation that followed Bill posted this

This is very close to the way in which I have started to think about software development in general and more specifically, the more scientifically rational approach to the engineering of software that I try to apply and promote.

For me these two ideas collide.

Software Development is an Act of Creativity

David Deutsch’s “Good Explanations” are deeper and more difficult than they sound. In striving for a “Good Explanation” we are required to gather information to allows us to “create knowledge”.

I describe software development as an inherently creative process. We don’t often consider it as such and much of software development is, incorrectly, treated as an exercise in production rather than creativity and suffers as a consequence. This misconception has dogged our industry and how we undertake the intensively creative task that is software development.

We are trying to create knowledge, in the form of a computer program, that captures our best understanding of the problem that we are trying to address. This is entirely a process of exploration and discovery. The encoding of the knowledge, in the form of something executable, is merely a transcription exercise. So the thinking, the design, the discovery of “good explanations” that fit our understanding is at the heart of all good software development.

Of course “merely a transcription exercise” underplays the complexity of that part of the process, but my point is that the technicalities of coding, the languages, the tools, the syntax of the instructions themselves have the same relationship to software development that maths does to physics. These things are tools that allow us to grow and extend our understanding. They are not the thing itself. Maths, and coding, are great fun. I completely understand, and recognise in myself, their appeal, but for me at least, that fun is enormously amplified when I can apply them to something practical. Ideally something that helps me deepen my understanding. Something that helps me to get to “better explanations”.

This is kind of obvious if we think in terms of computer science, but kind of missed in much of the discussion and practice that I observe in the software development community.

Software Development is Always a Process of Discovery

If we think back to our computer science studies we know that we only need a Turing machine, any Turing machine, to solve any classically computable problem. So the choice of tools, language, architecture, design are all only choices. These tools are not unimportant, but neither are they fundamental to solving any given problem.

I can write code to solve any computable problem in any language or paradigm. The only difference is how efficient I am in transcripting my ideas. Functional Programming, OO Programming, Ruby on Rails, C++, Java, Assembler can all only render the same ideas.

Of course it is a bit more complex than that. Certain programming approaches may help me to think, more easily, of some kinds of solution, others may hinder me. However, I believe that there is something deeper here that matters profoundly to the creation of good software.

It is the act of discovery and of learning, understanding the problem in more depth, that characterises our work and is the real value of what we do.

I believe that we should optimise our development approach, tools and processes to maximise our ability to foster that learning and process of discovery. We do this by creating a series of better and better explanations of the problem that we are attempting to solve, and the techniques (code) that we are employing to solve it.

Creating “Good Explanations”

Our “good explanations” take specific forms. They are the documentation and tests that describe a coherent picture of what our systems should do. They are the code that capture our best current theory of how our code should do the things it should. They are the ideas in our heads, the descriptions and stories that we tell each other, that allow us to understand, diagnose problems, and extend and maintain our systems. These are our good explanations and one of the profound advantages that we have over most disciplines is that we can make many of these “explanations” self-validating for consistency by automating them.

I have been a long-term adherent of Test Driven Development (TDD). I don’t take this stuff lightly and over the years of practicing it have refined my take on it. It is an old statement, not original to me, that TDD is not really about testing. I was peripherally involved in the birth of a thing called Behaviour Driven Development (BDD). The idea was to try and re-focus people’s thinking on what is really important in TDD. BDD was born as a means of teaching TDD in a way that led to the higher-value ideas of Behavioural focus and the use of “Executable Specifications” to drive the development of our software. It is a very effective approach and I teach it, and commend it, to the teams and organisations that I work with.

I now think that there is something more profound going on here though, and for me David Deutch’s “Good Explanations” hold the key. When we develop some software, any software for any purpose, we are, nearly always, embarking on a process of discovery.

We need to discover a lot of stuff. We need to learn more about the problem that our software is intended to address. We need to learn about what works for the consumers of our software, and what doesn’t. We need to discover what designs work well and give us the behaviours that we desire. We need to discover if our solutions are fast-enough, robust-enough, scalable-enough and secure-enough. We start out knowing little about all this, and begin learning from there. At any given moment, in the life of a software system, all of this stuff only adds up to “our best current theory”. We can never be certain of any of it.

Optimising for Learning

For the vast majority of human history we were really quite bad at learning. Then a few hundred years ago, we discovered how to do it. We call the trick that we learned then “Science”.

Science is humanity’s best, most effective approach to learning – Deutsch would say “gaining new knowledge”. Fundamental to this approach, according to Deutsch, is the formation of these “good explanations” and their defining characteristic that “they are hard to vary” without invalidating them.

In trying, at multiple levels, to capture a “good explanation” of what is going on. We are trying to describe the logic and algorithms that capture behaviours that we are interested in. We are trying to describe the data structures of the information that we deal with and process. We are trying, in some manner, to describe the need that our software is intended to address for our users or the market niche that our cool new idea is hoped to exploit.

All of these “descriptions” are “explanations” of our understanding. To transform these “explanations” into “good explanations” our “explanations” need to be more rigourous. The need to include everything that we know and, as far as we are able, check that our “explanation” fits all of the facts.

“Good Explanation” – Example

A good example of this, taken from Professor Deutsch’s book, is the idea of seasons. Some people believe that winter is caused by the Earth having an elliptical orbit and so being further from the Sun for part of the year. This is a good explanation in that I can’t vary it without changing it significantly. If the idea is correct, changing the explanation to say “The seasons are caused by Earth having a circular orbit” doesn’t work because that completely changes the explanation.

So this seems like a reasonable idea, and, even better, it is easily falsifiable. If this were true, if seasons are caused by the distance of the Earth from the Sun, then it should be winter at the same time of the year all over the planet, because the planet is in the same place in its orbit whether I am in London or Sydney. This isn’t the case, so this theory fails. It is a bad explanation because it doesn’t fit ALL of the facts.

Let’s try again. Observations show that for any given location on the Earth, the Sun will rise and set at different points on the horizon at different times of the year. Ancients, before global travel, knew this. A good explanation for this is that the axis of the Earth’s rotation is tilted with respect to its orbit around the Sun. The axis is tilted and precesses as the Earth orbits the Sun. That means that when our part of the planet is tilted toward the Sun we get more energy from the Sun because it is more directly overhead (we call this Summer) and when tilted away we get less energy (we call this Winter).

So if I was an ancient Greek, and knew about axial tilt as an explanation of seasons I could make a prediction. When it is Summer here, it will be Winter on the opposite side of the planet. This explanatory power is profound. It allows ancient Greeks to predict the seasons in a place that their descendants wouldn’t get to travel to for thousands of years!

Engineering – Applied Science

So what has all this philosophy of science stuff got to do with software? Well this science stuff is humanity’s best problem solving technique. It is the difference between essentially static, agrarian civilisations that lasted for tens of thousands of years with virtually no change and our modern, high-tech civilisation that doubles its knowledge every 13 months. The application of science to solving practical problems is how we solve the most difficult problems in the world. It is also what we call “Engineering”.

I believe that we should apply this kind of thinking, engineering thinking, to software development. What that takes is a significantly more disciplined approach to software development.

The rewards though are significant. It means that we can create high-quality software, more efficiently, more quickly than we have before. It means that our software will better meet the needs of our users and it means that the organisations in which we work can be more successful, while we are less stressed by trying to solve insoluble problems like “when will I be ready to release the new feature and get the product owner off my back?”.

So, step 1 is to approach software development as an exercise in learning, of discovery.

If our best way to learn is Science, and software development is all about learning, then we should apply the lessons of Science to our approach to software development.

An Engineering Discipline for Software

Following Deutsch’s model we should be trying to create “good explanations” that are “hard to vary” and then we should evaluate our explanations with each other, and with reality to confirm that they are consistent. What does this mean in practice?

We could try to write down some explanations of what we would like our software to achieve. We are not going to understand the totality of what we want our software to achieve at the outset, that is something that we will learn as we progress and understand the problem, and hopefully the demand, in more depth. So we are looking for a way in which we can capture our current intent and expectations in a form that we can later extend. How wonderful would it be if we could somehow capture these explantations of our current understanding in a form that would allow us to confirm that they are consistent with one another and met as we proceed to elaborate and extend our theories.

To me this is pretty much the definition of TDD. It allows us to record an incrementally evolving collection of, hard to vary, explanations that capture our current understanding. If we are smart, we capture them in a way that allows us, with the help of Continuous Integration, to immediately see if our theories, our “good explanations”, in the form of our code meet our expectations – do the tests pass?

This approach allows us to construct and re-use an automated system of checking that our “good explanations” are consistent with one another, that the body of our knowledge (of the system) as a whole is self-consistent. This, in turn, means that, as our understanding deepens, we can make small changes to our ideas and quickly and efficiently confirm that everything still makes sense. This approach allows us to stay informed about the state of our system-wide understanding, even as the scope of our system extends beyond our ability to intuitively understand it in its entirety. It means that we can extend and deepen our knowledge in a particular focused area (a new feature of the system).

I believe that the TDD approach, refined and elaborated upon by Continuous Delivery, represents a genuine “Engineering Discipline” for software development. I don’t mean this in a loose sense. I don’t mean that this is “analogous to Engineering”. I mean that it allows us to use a more scientifically rational approach to validating our ideas, measuring their effect and maintaining an ever increasing, consistent, collection of “good explanations” of our system and its behaviour.

Posted in Culture, Effective Practices, Engineering Discipline, Software Engineering | Tagged | Leave a comment

Hygiene Factors for Software Development

I got into a small debate about software development with someone recently via the comments section to a previous blog-post.

During the course of the debate I thought of an analogy to make part of my argument, but I think that it has broader applicability, which triggered this post.

I have been talking to a lot of people lately about “Software Engineering” and debating with people that I know, and some that I don’t, about what it takes to establish a profession, and an engineering discipline.

I perceive a reasonably broad consensus, amongst people that we may consider thought-leaders in our industry, some of whom I am happy to call friends, about what “good” software development looks like. I also perceive a level of dismay in that group about much common practice.

So what are these disciplines and where is the consensus?

I perceive a broad agreement that waterfall style thinking, although still very common in practice, is a busted idea. The data is in, it just doesn’t produce great software!

Software development is a learning process, from beginning to end. So we must work to establish effective, high-quality, fast feedback loops in order to maximise our opportunities to learn. That means working iteratively, as well as lots of other things.

We are not good at predicting the future and so we must be experimental, we must be sceptical of our ideas and find ways to evaluate them quickly and effectively. We need to be more data-driven, measuring rather than guessing.

Automated testing provides a substrate that helps us to achieve many of these goals. Taking a test-driven approach to development enhances the degree to which we can carry out these fast, cheap experiments in the design, and implementation, of our code.

If I am to be intellectually honest in my convictions, then all that I have just said about the development of code is also true about the creation and evolution of our approach to development. We should be data-driven, empirical, experimental in our approach to improving development process.

On the “data-driven” front we are making some progress. The excellent work done by my friends at DORA has raised the bar on measurement of process and practice in our industry. Their new book Accelerate explains the science behind their measurements. The results of these measurements are that, for the first time, we have data that says things like “Your company makes more money if you do x”, where ‘x’ is doing some of the things above.

The DORA folk have a model that predicts success (or failure) of your development approach. All of this is based on a peer-reviewed approach to data collection and analysis.

We can interpret these perceptions in several ways. Perhaps I am wrong and merely echoing the contents of my own filter-bubble (probably to some extent!). Most of the “thought leaders” that I am thinking of are old-hands, a polite euphemism meaning that my social group is getting-on a bit. Maybe these are the rants of old men and women (though most are men, which is another problem for our industry sadly).

A more positive interpretation, and one that I am going to assume for the rest of this post, is that this represents something more. Perhaps we are beginning to perceive the need to grow-up, a little, as an industry?

My own, primary, interest in this is around the engineering disciplines that I think that we should try to establish as a norm for software developers who consider themselves professionals. I would like us to have a more precise definition of what “Software Engineering” means. It would need to rule some things out, as well as define some things that we should always do.

Others are interested more in the “Profession” side of things. I have recently seen a rise in people discussing ideas like “ethics” in software development. Bob Martin has a couple of interesting talks on this, and closely related, topics. He makes good points about the explosive growth of our industry and the consequent dilution of expertise. He estimates that the average level of experience, amongst software developers, is just 5 years. As a result we, as an industry, are very bad at learning from the mistakes of the past.

I have been careful in my choice of words here. Currently we are not a “Profession” we are a “Trade”. The difference between these two is that a “profession” demands qualifications as a barrier to entry, and has rules to reject people that don’t conform to its agreed, established norms. By these defining characteristics we don’t qualify as a profession.

You can’t practice law or medicine without the appropriate qualifications. In our industry, if you can pass the interview, you can take part. If I can convince an interviewer that I am competent, over a small number of hours during the course of an interview, I could go and write software that controls an aeroplane, a medical scanner or a nuclear power plant. An individual company may have rules that demand a specific degree, or other qualification, but our “trade” does not.

If you are a surgeon and you decide that washing your hands between operations is a waste of your valuable time, once people notice of the increased death-rate at your hands, you will be “struck-off” and not allowed to practice surgery ever again, anywhere.

There can be no profession without professional discipline.

In 1847 Ignaz Semmelweis made an important discovery:

“The introduction of anaesthetics encouraged more surgery, which inadvertently caused more, dangerous, post-operative infections in patients. The concept of infection was unknown until relatively modern times. The first progress in combating infection was made in 1847 by the Hungarian doctor Ignaz Semmelweis who noticed that medical students fresh from the dissecting room were causing excess maternal death compared to midwives. Semmelweis, despite ridicule and opposition, introduced compulsory hand-washing for everyone entering the maternal wards and was rewarded with a plunge in maternal and foetal deaths, however the Royal Society dismissed his advice.” (Wikipedia

This resonates with me. I advocate for some specific practices around software development. These practices work together, in sometimes subtle ways. I believe that the combination of these practices provide a framework, a structure, a disciplined approach to software development that has the hallmarks of a genuine “engineering discipline”.

I believe that, like “washing your hands” as a surgeon, some of these disciplines are so important that they should become norms for our industry. I don’t doubt that you can write software without fast feedback, without automated tests, without an experimental approach, without collaborative teams and with big-up-front designs and with a 12 month plan. A positive outcome, though, is much less certain. Just because some surgeons had patients that survived, despite their lack of hygiene, doesn’t mean that hygiene isn’t a better approach.

These days, nobody can consider themselves a surgeon if they ignore the disciplines of their profession. I believe that one day, one way or another, we will, of necessity, adopt a similar approach.

If we are to establish ourselves as a profession, rather than as a trade, we will need to do something like this. Software is important in the world. It is the revolutionary force behind our civilisation at the moment. I foresee three futures for our industry.

1. We do nothing. At some point, something REALLY bad happens. Some software kills LOTS of people, or maybe destabilises our political, economic or social institutions. Regulators will regulate and effectively close us down, because they will get it wrong. (It has taken us decades to understand what works and what doesn’t, and we are supposed to be the experts!)

2. We start trying to define what it means to be a “Software Professional” in the true sense of the words. Something bad happens, but the regulators work with us to beef-up our profession, because they can see that we have been trying to apply some “duty of care”.

3. The AI Singularity happens and our Silicon overlords take the task of writing software out of our hands.
Ignoring 3 for now…

Scenarios 1 and 2 are both problematic.

I fear that we will continue with 1. The short-term economic imperative will continue to drive us, for a while, until the population at large realise just how important software has become. At which point there will be repercussions as they react to the lack of a sufficient duty-of-care in many instances. The VW emissions scandal is an early warning of this kind of societal reaction, I think.

Scenario 2 is problematic for different reasons. I think that it is the more sensible strategy, but it demands that we change our industry and allow it to progress from trade to profession. Daunting! At which point, if we succeeded, I would be expelled for not having any relevant qualifications. This is a big challenge, and not just for me personally ;-). Our industry is still growing explosively, educational establishments are not really delivering people with the skills ready to be “professional” in the sense that I mean. Many universities (maybe even most) still teach waterfall development practices for goodness sake!

My own experience of hiring and training young people into our industry suggests that there is relatively little advantage in hiring Computer Science graduates over most other graduates. We pretty much had to start from scratch with their brain-washing, errrr “on-the-job training”, in both cases. It is easy, even common, to graduate from a CS course and not be able to program, let alone program well. Physics, and other hard-science, graduates have a better understanding of experimental discipline and scientific rigour. The main problem with physicists (and most CS graduates) is getting them to realise that “yes, programming is actually quite difficult to do well” and the techniques that work for a few lines of private code don’t scale well.

There is still much debate to be had. Despite the fairly broad consensus that I perceive on what it means to apply “engineering thinking” in software, I still regularly get people arguing against the practices that I recommend. If I am honest, most of these arguments are ones that I have heard many times. Often these arguments are based on dogma rather than measurement or evidence. If we are to be more scientific, apply more engineering discipline to our work, we cannot base our decisions on merely anecdote. That is not how science and engineering work!

I am not arrogant enough to assume that I have all of the answers. However, I confess that I am hubristic enough to believe that the people expressing “ridicule and opposition” on the basis of dogma or only anecdote don’t have a strong case. Mentally I dismiss those arguments as being analogous to the surgeons who don’t “wash their hands”.

If you want to change my mind, change it with data, change it with evidence.

I think that we are in the same state as surgeons in the 1850s. Today, there is no reputable surgeon in the world that does not wash their hands before surgery now. This discipline wasn’t always obvious though. I believe that we have identified a number of practices that are the equivalent for software development of “washing your hands” for surgeons. I spend a lot of my time describing these despite <occasional> “ridicule and opposition” 😉

In both cases, existing practitioners, who don’t “wash their hands”, claim that this is unnecessary and a waste of time. I think that the data, and, I hope one day, history, is on my side.

Posted in Agile Development, Culture, Effective Practices, Engineering Discipline, Software Engineering | 1 Comment

Perceived Barriers to Trunk Based Development

A friend of mine has recently started work at a new company. She asked me if I’d answer a few questions from their dev team, so here is the second…

Q: “Currently at MarketInvoice we use short-lived feature branches that are merged to master post-code review. How would you recommend we shift towards trunk based development and are there any other practises you would recommend to reduce/eliminate the bottleneck of code review?”

I perceive three barriers to the adoption of trunk-based-development in the teams that I work with…

  • The need for Code Review.
  • A cultural assumption that you only commit (to master/trunk) when work is complete.
  • A lack of confidence in automated tests.

Code Reviews

I think that code-review is a very useful practice. We get good feedback on the quality of our work, we may get some new ideas that we hadn’t thought of, we are forced to justify our thinking to someone else, and, if we work in a regulated industry, we get to say that our code was checked by someone else.

All of these are good things, but we can get them all, and more, if we adopt pair-programming.

Code review is great, but it happens when we think that we have finished. That is a bit too late to find out that we could have done better. From a feedback perspective, it would be much more effective if we could find out that an idea, or approach, could be improved before, or immediately after, we have written the code rather than after we thought we had finished. Pair programming means that we get that feedback close to the point when it is most valuable.

Pair programming is a code-review, and so satisfies the regulatory need for our changes to be checked by someone else, at least is has in every regulatory regime that I have seen. Pair programming is also much more than just a continual review. One way to look at it is that we get the code-review as a side-benefit, for free.

This means that the review does not impose any delay on the development. The code is being reviewed as it is written and so the review is both more thorough and adds no additional time to the development process.

So, my first answer is… Pair Programming!

Don’t wait to commit

This is a mind-set thing, and makes perfect sense. It seems very logical to assume that the ideal time to commit our changes is when we think that they are ready for use – the feature that we are working on is complete.

I think it is a bit more complicated than that though. I describe this in more detail in my post on “Continuous Integration and Feature Branching

If we want the benefits of Continuous Integration we need to commit more frequently than when we think that we are finished. The only definitive point at which we can evaluate our changes is when we evaluate them with the “production version” of our code which is represented by trunk (or master). CI on a branch is not CI! It is neither integration, at least not with the version of the code that will be deployed into production, nor is it continuous because you only integrate, with the version of the code that is deployed into production, when the feature is “finished”.

So to practice Continuous Integration, which is a pre-requisite for Continuous Delivery, we have to commit more frequently to the copy of code destined for production and so we must change our working practices.

This is a big shift for some people. It is probably one of the most profound shifts of mind-set for a developer in the adoption of Continuous Delivery. “What, you want me to commit changes before I am finished?” – Yes!

Continuous Delivery is defined by working in a way so that your software is in a releasable state after every commit. That doesn’t mean that all of the code needs to be useful. It just means that it “works” and doesn’t break anything else.

In the language of Continuous Delivery we aim to “separate deployment from release”. We can deploy small, simple, safe changes into production and only “release” a feature when all of those small changes add up to something useful.

This leads us into the territory of a much more evolutionary approach to design. Instead of thinking about everything up front, even for a small feature, we will work in a fine-grained, iterative way that allows us to try ideas and discard them if necessary on the route towards something that works better.

This has lots of good side-effects. Not least it means that I will design my code to allow me to change my mind and get things wrong without wasting all of my work. That means that my code will have good separation of concerns, be modular and will use abstractions to hide the details of one part of my design from others. All of these are hallmarks of high-quality code. So by working more incrementally, I get higher quality designs.

Automated Testing

“I can’t commit to trunk before I am finished because I may break something”. To me, that speaks of a lack of confidence in testing and/or a very traditional mind-set when it comes to testing strategy.

It kind of assumes that you can’t test your feature until it is finished. I think that that is old-school thinking. This is a problem that we know how to solve – “Test First!”.

This problem in part stems from the language that we have chosen to describe the use of automation to verify that our code works. We call these things “Tests” which tends to make us thing of performing this verification as a final step before we release. I wonder if the adoption of a “test-first” approach would have been different if we had called these things “specifications” rather than tests. “Specify first” seems more obvious perhaps than “test first”.

If we see our automated evaluations as “specifications” that define the behaviour that we want of our systems, we must obviously do the thinking, and create the automated version of these specifications, before we start to meet them by building code.

By building software to meet executable specifications of its behaviour we eliminate whole classes of errors, but even more importantly, we drive the design of our systems towards higher-quality. I argue this in an earlier post on “Test Driven Development“. The properties of code that make it testable are the same properties that we value in “high quality code”.

I have worked on large-scale complex systems where we could release at any time without fear of breaking things because our automated testing caught the vast majority of defects. Employing tests as “executable specifications” which describe the desired behaviours of our systems has a dramatic impact on the quality of the code that we produce.

In a study of production defects the authors estimated that over 70% of production defects would be eliminated by a more disciplined use of automated testing.

Using a test-first approach drives quality into our designs, protects against the most common causes of production defects and allow us to move forwards with more confidence.

Posted in Agile Development, Continuous Delivery, Culture, Effective Practices, Feature Branching, Pair Programming, TDD | 3 Comments

Pair Programming for Introverts

A friend of mine has recently started work at a new company. She asked me if I’d answer a few questions from their dev team, so here is the first in a short series of their questions and my answers…

Q: “Pair programming has been shown to increase quality and reduce overall development time. Nevertheless, some need heads down focused time on a problem. How do you balance this?”

My preference is to strongly encourage teams to adopt the norm that most work will be done working in pairs, but not to make it a rule. I think it right to leave room for people to decide for themselves when it doesn’t make sense.

However, you are right, ALL of the data that I have seen from studies of pair programming say that it produces higher-quality output, and so in the long run, is significantly more efficient in delivering new code. More than that, I know of no better way to encourage collaboration, learning and continual improvement in a team than pair programming.

(Links to some of that research at the end of my blog post “Pair Programming – The Most Extreme XP Practice”)

So it is strongly in a team’s interest to adopt and encourage pair programming as the norm. It is not good enough to reject it because some people don’t like it. That would be like mountain rescue teams rejecting the use of ropes because it is annoying to carry them up the hill. Some things have value even if they take some work.

For me, this means that it is worth some effort, maybe even significant effort, for a team to adopt, learn and make pair programming a fundamental part of their development culture.

My experience has been that most people, before they have experienced it, are nervous of pairing.

In part I think that this is a cultural thing, we “program” people to imagine software development as a lonely introspective act. I don’t think that good software development is really like that. It is, at its heart, a process of learning.

We learn best when we can try-out new ideas and quickly discard the bad ones. One way to test ideas is to bounce them off another person. So pair programming provides us with a mechanism to quickly and cheaply exercise ideas and weed out some of the bad ones.

There are also some individuals who will always find pair programming stressful.

If I am honest, I believe that these individuals have a more limited value to the team. They may have value, but that value can’t be as much as someone of similar skill who learns faster and teaches more.

Introverted people are more sensitive to stimulation than others, and so need more quiet time to reduce the cognitive clutter. I am one of these people. I need, periodically, to be on my own to organise my thoughts. This doesn’t mean that people like this can’t take part in pair programming, it does mean that you have to give them some space, some of the time.

So, my idea of “optimal” is to do most, nearly all, development work in pairs but allow humans to be human. If someone needs time to form their thoughts, or learn some tricky concept alone, or just needs some quiet time to recharge for a bit, give them that time.

There is another important aspect to this. There is some skill to pair programming. It takes time to learn some of the social aspects. For example, one very common behaviour that I see, in newbies, is for my pair, when I am typing, telling me letter-by-letter when I make a typo or what the instruction is. They are trying to be helpful, but they are not.

Watch your own typing for a bit. If you are anything like me, then your typing will progress forwards and backwards as you make little mistakes and then correct them all of the time. When this happens you know, as you type, that you made a mistake. Most errors you correct immediately. Someone telling you at this point, actually slows you down. It interrupts the flow of your thinking – and it is irritating.

So when you are pairing, and you are not typing, give people a chance to spot, and correct, their own mistakes. Only mention a typo when the typist has moved on and clearly missed it. Only mention the correct use of a language construct or api call if the typist is clearly stuck. Otherwise KEEP QUIET!

The classic description of the roles in pair programming are “Driver” (the person who is typing) and “Navigator” (the person who is not). This is a bit crude, but close. If you aren’t typing your focus should be on the direction of the design rather than the typing.

The other important aspect of pair programming as a learning activity is to regularly rotate the pairs. Change pairs often, don’t allow pairs to become stale. My preference is to change pairs every day.

This sounds extreme to some people. It means that nearly everyone works on nearly everything that the team produces over the period of a week or two. It means that you get to see different people’s styles of working (and pairing) and learn from them. It means that you get to work with the person on the team that you find trickiest to pair with and with the person that you enjoy working with the most, on a regular basis.

Pairing means that you are working in very close proximity to other people. Think of your pair as a team, you have shared goals and will succeed, or fail, together. Be considerate, be collaborative, be kind!

If you get this kind of stuff right, then the barriers to pair programming begin to reduce. Even the introverts on your team will not only take part, but will benefit from it.

Pair programming takes time to adjust to. This is not something that you can try for a day or two. It takes a while for a team to get really good at it, so allow yourselves the time, don’t give up too soon.

Posted in Agile Development, Culture, Effective Practices, Pair Programming | 1 Comment

CI and the Change Log

I get in to debates about the relative merits of “Continuous Integration (and Delivery)” vs those of “Feature Branching”  on a fairly regular basis.

A common push-back against CI, from the feature-branchers, is “you can’t maintain a clean change-log”.

I guess this depends on how important you think the change-log is and what it is for.

Is the change-log equally, or more important than working software? Of course not!

I know that statement is a bit extreme but it is *kind-of* a relevant question. CI is a practice that comes with some trade-offs, but it is the best way that we have discovered of maintaining our software in a working state so far.

Analysis from the “2017 State of DevOps Report” found the following:

“High performers have the shortest integration times and branchlifetimes, with branch life and integration typically lasting hours.

Low performers have the longest integration times and branch lifetimes, with branch life and integration typically lasting days.

These differences are statistically significant.”

The VCS change log tells a story, but what is the story and what is it for?

If I connect my “story/requirements management system” (JIRA etc) to my VCS via a tag in the commit message, I can trace every commit to a story. So I have traceability. So I guess the next question is what are the use-cases for a change-log?

I can think of two broad groups of usage for a change-log:

1) Some kind of audit-trail of changes, maybe useful for a regulator or compliance person to see the history of changes.

2) An index of changes that a developer can use to navigate the history.

If I adopt CI, and make fine-grained, regular commits, each of them commented on and linked to a story (or bug), then I can tell the story of the story. I have my audit trail. It will be very detailed. It may even wander around a bit “Make the button blue” and later “Make the button green” but that was the true story of the development. This is a good, accurate representation of the life of the change.

I know that each commit was related to the story, so from the perspective of an auditor I have a definitive, albeit granular, statement.

From the perspective of a developer wanting to know what change did what, I have a more detailed picture that too, because of this more granular reporting. I can build up the story, in fine detail of the evolution of the ideas. I have not lost anything, I have more information not less. The picture may be a bit messier, but that only represents the reality of the evolution of the design.

I confess that I don’t really understand the desire for a “clean change log”. What does that mean? It seems to me to imply an assumption that once I have finished a “Story” I am done.

What is the difference between me playing “Story 1”, which “makes the Button green” and later “Story 5” which “makes the button blue” and me changing my mind in the midst of  “Story 3” and making the same change?

I think that this desire for a “clean change-log” may be based on an illusion of software development as an ever increasing collection of desirable features rather than as an exploration of a problem-space. I think that development is much messier than that. It is much more the latter than the former.

If we are not learning-as-we-go that some of our ideas are wrong, we are not doing a very good job of software development. In my world, however granular or not, the idea of a “clean change-log” is an illusion.

I don’t believe that software development is like that. However I work, I am going to be returning to the code over and over again and refining and updating it as requirements are added and as my understanding evolves. So even if I have a log entry-per commit, I still need to read them all to know the state of the system at any given point, the only difference is one of granularity.

I am increasingly starting to view the collection of a fine-grained picture of the changes in our development process as an asset, not as a liability. Instead of thinking of the change-log as a linear record, think of it as part of the “historical search-space” of information, linked by keys (like the id of your story and the id of your release candidates), that you can navigate to build any picture you like of what happened. To my mind that is a more powerful tool, not a less powerful one.

Posted in Agile Development, Continuous Delivery, Continuous Integration, Effective Practices, Feature Branching | 5 Comments

Three Distinct Mind-sets in TDD

I have blogged about TDD before. I think that it is one of the most important tools in improving the design of our software, as well as increasing the quality of the systems that we create. TDD provides valuable, fine-grained feedback as we evolve the solutions to the problems that our code is meant to address.

Oh yes, and as a side-benefit, you get some nice efficient, loosely coupled, tests that you can use to find regression problems in future. 😉

I sometimes teach people how to practice TDD more effectively, and one of the things that I notice is that one subtlety that people often miss is the difference in focus for each of the TDD steps.

True TDD is very simple, it is “RED, GREEN, REFACTOR“.

  • We write a test, run it and see it fail (RED).
  • We write the minimum code to make it pass, run it and see it pass (GREEN).
  • We refactor the code, and the test, to make them as clean, expressive, elegant and simple as we can imagine (REFACTOR).

These steps are important not just as a teaching aid, but also because they represent three distinct phases in the design of our code. We should be thinking differently during each of these steps…


We should be wholly focussed on expressing the behavioural need that we would like our code to address. At this point we should be concentrating only on the public interface to our code. That is what we are designing at this point, nothing else.

If you are thinking about how you will implement this method or class, you are thinking of the wrong things. Instead, think only about how to write a nice clear test that captures just what you would like your code to do.

This is a great opportunity to design the public interface to your code. By focusing on making the test simple to write, it means that if ideas are easy to express in our test, they will also be easy to express when someone, even you in future, uses your code. What you are really doing, at the point when you strive for a simple, clear test, is designing a clean, simple to use, easy to understand API.

Treat this as a distinct, separate step from designing the internal workings of the code. Concentrate only on describing the desired behaviour in the test as clearly as you can.


Experienced TDD practitioners, like me, will tell you to do the simplest thing that makes the test pass. Even if that simple thing is trivial, or even naive. The reason that we advise this is because your code is currently broken, the test is failing. You are at an unstable point in the development.

If you start to try and do more complex things at this point, like make your design elegant or performant or more general, you can easily get lost and get stuck in a broken state for a while.

If the “simplest thing” is to return a hard-coded value, hard-code it!

This does a couple of things. It forces you to work in tiny steps, a good thing, and it also prompts you to write more tests that allow you to expand the logic of your code, another good thing.

Your tests should grow to form a “behavioural specification” for your code. Adopting the discipline of only writing production code when you have a failing test helps you to better elaborate and evolve that specification.

Don’t worry, we won’t forget to tidy-up the dumb, overly simplistic things that we do at this point.

Over-complicating the solution is one of the commonest mistakes that I see TDD beginners make. They try to capture too much in one step. They prefer to have fewer more complex tests than many, small, simple tests that prod and probe at the behaviour of their system. The small steps, in thinking and in code, help a lot. Don’t be afraid of many small simple tests.


Always refactor on a passing build. Wait until you are in the “GREEN” state before you begin. This keeps you honest and stops you wandering off into the weeds and getting lost! Make small simple steps and then re-run the tests to confirm that everything still works.

Refactoring is not just an afterthought, it is not just about aligning the indents and optimising the imports. This is an opportunity to think a bit more strategically about your design.

It is important that we treat it as a separate step. I often see things that I want to change either when writing a test (RED) or when writing code to make the test pass (GREEN). On my good days, I remember that this is not the time. I make a note and come back to it once the test is passing. On my bad days I often end up making mistakes, trying to do things in steps that are too big an complicated, rather than small and simple, and so I end up having to revert or at least think a lot harder than I need.

If you use a distributed VCS like GIT, I recommend that after each refactoring step, after you have checked that the tests all pass, commit the change. The code is working, and the committed version gives you a chance to step-back to a stable state if you wander-off into more complex changes by mistake.

In general, I tend to commit locally after each individual refactoring step, and push to origin/master after finishing refactoring, but before moving-on to the next test.

Another beginner mistake that I frequently observe is to skip the refactor step all together. This is a big mistake! The refactor step is the time to think a little bit more strategically. Pause and think about the direction in which your code is evolving, try and shape the code to match this direction. Look for the cues that tell you that your code is doing too much or is too tightly-coupled to surrounding code.

One of my driving principles in design is “separation of concerns” if your code is doing “something AND something else” it is wrong. If your code is doing a business level calculation and is responsible for storing the results – wrong! These are separate and distinct concerns. Tease out new classes, new abstractions that allow you to deal with concerns independently. This naturally leads you down the path towards more modular, more compose-able designs. Use the refactoring step to look for the little cues in your code that indicates these problems.

If the set-up of your tests is too complex, your code probably has poor separation of concerns and may be too tightly-coupled to other things. If you need to include too many other classes to test your code, perhaps your code is not very cohesive.

Practice a pause for refactoring every single time you have a passing test. Always look and reflect “could I do this better?” even if sometimes the answer is “no it is fine”.

The three phases of TDD are distinct and your mental focus should also be distinct to maximise the benefit of each phase.

Posted in Continuous Integration, Effective Practices, Software Design, TDD | 10 Comments

A Few Thoughts on Feature Flags

I confess that “Feature Flags” make me a bit nervous. Despite this I think them a useful and important tool in our ability to achieve Continuous Integration.

So why do they make me nervous? Well, they are a form of “branching” they are designed to isolate change and as I have described previously, I think that branching works against Continuous Integration.

There is a big difference between Feature Flags and VCS-based branches though. Feature Flags isolate at the level of behaviour rather than at the level of code. This is an important and valuable distinction.

“Feature Flags isolate at the level of behaviour rather than at the level of code”

One of my practical objections to the use of VCS branches for normal development is that they place a barrier to promiscuous refactoring. All of the best code-bases that I have worked in had a high churn-rate. We would change them often and make them better in small ways all the time. Branches tend to prevent us from doing that, Feature Flags, on the other hand, allow it.

The danger with Feature Flags is that they can introduce considerable complexity. Which version of your code do your test? Feature on or off? Both?

If “both”? you are on a journey into exponential complexity growth as you add more flags – Flag “A” on Flag “B” on, Flag “A” on Flag “B” off, and so on! This is a never-ending game that you can’t win in any definitive way.

I tend to employ a hierarchy of approaches to allow me to make progressive changes in my code running under Continuous Integration.

First, I prefer to release change directly, make a change and have people use it.

Next, I will use “dark-release” or “branch-by-abstraction”. Dark-release allows me to build up, and test, stuff that people aren’t using yet. Branch-by-abstraction encourages me to create abstractions in my code. These abstractions allow me to switch the implementation of these abstract features easily. It also fits with my style of coding where I care very much about separation-of-concerns and abstraction. Branch-by-abstraction can even allow me to run the old and new versions of a feature in parallel! This opens another world of possibilities for measuring the merits, or otherwise, of new features.

Only if none of these work will I use Feature Flags. This is largely because of the testing problem. For the types of systems that I have worked on for the past few years, I want to test what is running in production.

Another facet of this is that, on the whole, I prefer to make my Deployment Pipeline so efficient, that if I want to change the config of my system, even its Feature Flags, I will push the change through the pipeline, and so I can test it before release!

Posted in Agile Development, Continuous Integration, Effective Practices | 7 Comments

Continuous Integration and Feature Branching

Recently I spoke at the Pipeline Conference in London. I gave a talk on “Optimising Continuous Delivery” to a bunch of people who were self-selected as interested in Continuous Delivery, most of them would already consider themselves CD practitioners, Pipeline is a conference dedicated to learning about CD!

Early in my talk I described some of the ground-rules for CD, those practices that I consider “table-stakes” for playing. One of my slides is intentionally slightly jokey. It describes my advice for branching strategies in the context of CD.


I knew that this would be contentious, it is always contentious. This is the practice that I advise that I get most push-back on. EVERY TIME!

Before I had got home from the conference my twitter account ‘@davefarley77’ had gone a bit mad. Lots and lots of posts, for and against, questions and challenges and while a week later it has slowed down a bit, the rumblings continue.

I wrote about feature branching some years ago. I was about to say that the debate has moved on, but in reality I don’t think that it has. The same questions and issues arise. So this blog post is meant to add a few more thoughts on the topic.

The push-back that I get when I express my view that any form of branching is counter, in principle, to the ideas of Continuous Integration is varied.

At one extreme I get “Heretic, burn him at the stake” kind of feedback, at the other “Yes, but it can’t possibly work without Feature Branching – you must work in small teams and/or on trivially simple projects”.

The first point is fair enough, I am, constitutionally, a heretic. I like to be skeptical about “received wisdom” and question it.

In this case though, my views on branching are from experience rather than mere academic skepticism. I have been a professional software developer for nearly four decades now. I have tried most things over the years. I have refined my understanding of what works and what doesn’t on a lot of projects, trying a lot of different tools, technologies, methodologies and techniques.

In response to the second class of “push-back” I do sometimes work in small teams, but also with some of the biggest companies in the world. For the past three decades I think that it is fair to categorise most of my development work as at the more complex end of the scale. Which is one of the reasons that I take some of these disciplines quite so seriously.

I am an adherent of agile principles and take them to their extreme with my flavour of Continuous Delivery when I am in a position to decide, or influence the decision.

I first practiced a version of Continuous Integration in 1991. We had a continual rolling build, a home built version control system, written in shell-script, and even a few simple “unit tests” on our C++ project. This pre-dated the popularity of CI by a considerable margin, but it worked really well!

What I learned on this project, and on many others, small and MASSIVE, is that what really matters is feedback! Fast and high-quality. The longer that you defer feedback, the greater the risk that something unexpected, and usually bad, will happen.

This is one of the ideas that inspired the evolution from Continuous Integration to Continuous Delivery. We wanted better feedback, greater insight, into the effect of our changes, whatever their nature.

So you can tell, I am a believer in, and advocate for, Continuous Integration. We create better code when we get fast feedback on our changes all of the time.

CI is a publication based approach to development. It allows me to publish my ideas to the rest of my team and see the impact of them on others. It also alows the rest of my team to see, as it is evolving, the direction of my thinking. When teams practice CI what they get is the opportunity to “Fail Fast”. If something is a problem, they will spot it REALLY quickly, usually within a handful of minutes.

CI works best when publications/commits are frequent. We CI practitioners actively encourage commits multiple times per day. When I am working well, I am usually committing every 15 minutes or so. I practice TDD and so “Red-Green-Refactor-Commit” is my mantra.

This frequency doesn’t change with the complexity of the code or size of the team. It may change with how clearly I am thinking about the problem or with the maturity of the team and their level of commitment to CI.

What I mean by that, is that once bitten by the feedback bug, you will work VERY hard to feed your habit. If your build is too slow, work to speed it up. If your tests are too slow, write better tests. If your hardware is too slow on your build machines, buy bigger boxes! I have worked on teams on some very large codebases, with complex technologies that still managed to get the fast feedback that we needed to do high-quality work!

If you care enough, if you think this important enough, you can get feedback fast enough, whatever your scale! It is not always easy, but it has always been possible in every case that I have seen so far – including some difficult, challenging tech and some VERY large builds and test suites!

“What has all of this got to do with branching?” I hear you ask. Well if CI is about exposing our changes as frequently as possible, so that we can get great feedback on our ideas, branching, any form of branching, is about isolating change. A branch is, by-design, intended to hide change in one part of the code from other developers. It is antithetical to CI, the clue is in the name “CONTINUOUS INTEGRATION”!

To some degree this isolation may not matter too much. If you branch, but your branch is VERY short-lived, you may be able to get the benefits of CI. There are a couple of problems with this though. First, that this is not what most teams do. Most teams don’t merge their branch until the “feature” that they are working on is complete. This is called “Feature Branching”.

Feature Branching is very nice from the perspective of an individual developer, but sub-optimal from the perspective of a team. We would all like to be able to ignore what everyone else is doing and get on with our work. Unfortunately code isn’t like that. Even in very well factored code-bases with beautiful separation-of-concerns and wonderfully loosely-coupled components, some changes affect other parts of the system.

I am not naive enough to assert that Feature Branching can never work, you can make anything work if you try hard and are lucky. Even waterfall projects occasionally produced some software! My assertion is that feature branching is higher-risk and, at the limit, a less efficient approach.


The diagram above shows several paths from idea to working software in production. So if we want effective, high-quality feedback where in this diagram should we evaluate our changes? Point 1 is clearly no good, the changes on the branches, 5, 6 and 7, are never evaluated.

We could evaluate the changes after every merge to trunk, 2, 3 and 4. This is what lots of Feature branching teams do. The problem now is twofold:

1) We get no feedback on the quality of our work until we think that we are finished – Too Late!
2) We have zero visibility of what is happening on other branches and so our work may not merge. – Too Risky!

Before the HP Laserjet Firmware team made their move to Continuous Delivery, their global development team spent 5 times as much effort on merging changes between branches as on developing new features!

(See from time 47:16 in this presentation  also “A Practical Approach To Large Scale Agile Development”)

At this point my branch-obsessed interlocutors say “Yes, but merging is nearly free with modern tools”.

It is true! Modern distributed Version Control Systems, like GIT, have very good merge tools. They can only go so far though. Modern merge tools are good at the optimistic lock strategy of deferring locking things down until you see a conflict, at which point they request some help, your help. Most of the time merges are simple and automatic, but often enough, they are not.

As soon as you need to intervene in a merge there is a cost and until the point of merging you don’t know how big that cost will be. Ever got to merge some change that you have worked on for several days or a week, only to find that the differences are so great that you can’t merge? Lots of teams do find themselves in this position from time to time.

Back to our diagram. What feature branch teams sometimes do is run a dual CI system, they run CI on the branches AND after the merge to Trunk. This is certainly safer, but it is also slow.

As ever, the definitive point is the testing that happens at the point of merge to Trunk. It is only at this point that you can honestly say “Yes, my change works with everyone else’s.”. Before that, you are hoping that someone else hasn’t done something horrid on another branch that breaks your stuff when you merge.

This approach is safer because you are getting some feedback sooner, from the CI running on your feature branch, but this branch is telling lies. It is not the real story. This is not a change set that will ever make it into production, it isn’t integrated with other branches yet. So even if all your tests pass on this branch, some may fail when you merge. It is slow because you are now building and running everything at least twice for a given commit.

The real, authoritative feedback happens when you evaluate the set of changes, post merge, that will be deployed into production, until your branch is finished and merged onto Trunk, everything else is a guess.

CI advocates advise working on Trunk all the time. If you want to be pedantic, then yes, your local copy of the code is a form of branch, but what we mean by “all the time” is that we are going to make changes in tiny steps. Each change is itself atomic and leaves the code in a working state, meaning that the code continues to work and deliver value. We will usually commit many of these atomic changes every day. This often means that we are happy to deploy changes into production that are not yet complete, but don’t break anything!

CI, like every other engineering practice, comes with some compromises. It means that we are only allowed to commit changes that keep the system working. We NEVER intentionally commit a change that we know doesn’t work. If we do break something the build stops and rejects our change, that is the heart of CI.

This means that we have to grow our features over multiple commits, if we want regular, fast, authoritative feedback. This, in turn, changes the way that we go about designing our features. It feels more like we “grow” them through a sequence of commits rather than take them aside, design them and build them in isolation and then merge them.

This is a pretty big difference. I think that this is one of the reasons for the second category of push-backs that I tend to get from people who are more used to using branches.

Q: “Yes, but how do you make anything complex in 15 minutes?” 

A: You don’t, you break complex things into a series of small, simple changes.

Q: “But how can a team fix bugs in production?”

A: They feed the fixes in to the stream of commits, like any other change to the system.

Q: “Ah yes, but how do you do code reviews?”

A: Pair Programming is my preferred approach. You get better code reviews and much more.

Q: “Ah, but you can’t do this for software like XXX or using technology like YYY”

A: I have build systems-software, messaging systems, clustering systems, large volume data-base backed systems, whole enterprise systems, some of the highest performing trading software in the world, as well as web-sites, games and pretty much any other type of software that you can think of using this approach.

I have done it in Java, C#, C++, Python, Ruby, Javascript, shell-script, FPGA systems, Embedded software and COBOL. I have seen other teams using this approach on an even wider variety of technologies and products. I think it works!

CI is about speed and clarity of feedback. We want a definitive picture of the quality of our work, which means that we must evaluate, precisely, the code that will go into production. Anything else is guessing. We want our feedback fast and so we will optimise for that. We work to position the machinery that provides that feedback so that it can try our changes destined for production as soon as possible, that is, as close to the point that we made the changes as we can achieve.

Finding our that my code is stupid or broken within 2 minutes of typing it is very different to having to wait, even as short-a-time as an hour for that insight. It changes the way that I work. I can proceed faster, with more confidence and, when I do mess-up, I can step back with very little cost.

So we want definitive feedback fast. That means that anything that hides change gets in the way and slows us down. Any form of branching is antithetical to Continuous Integration.

If your branch lasts less than a day, my argument against it is weakened, but in that case I will pose the question “why bother with branches?”.

I work on Trunk, “master” in my GIT repos. I commit to master locally and push immediately, when I am networked, to my central master repo where CI runs. That’s it!

I do compromise the way that I work to achieve this. I use branch by abstraction, dark-releasing and sometimes feature-flags. What I get in return is fast, definitive (at least to the quality of my testing) feedback.

Last years “State of DevOps Report” claimed that my style of development is a defining characteristic of “High Performing Teams”. If you are not merging your changes to Trunk at least daily, it predicts that your outcomes are more closely aligned with “Lower Performing Teams”.

There is a lot more to this, CI is not a naive approach it is well-thought out and very widely practiced in some of the most successful companies in the world. Trunk-based development is a core practice to CI and CD, it really is very difficult to achieve all of the benefits of CI or CD in the absence of Trunk-based development. You can read more about these ideas on the excellent Trunk-Based-Development site.

Posted in Agile Development, Continuous Delivery, Continuous Integration, Effective Practices, Feature Branching | 75 Comments

Answers to GOTO Cph 2017 Questions

I gave a presentation on my recommended approach to Acceptance Testing today, here at GOTO Copenhagen.

You can see an earlier version of this talk, from another conference here. GOTO will be publishing their version soon 😉

I ran out of time for questions, but here are my answers to questions submitted via the GOTO app…

Question: Those slides could use a designers touch, though 🙂
Answer: Fair enough, I am not a designer 😉
Question: How do we obtain repeatable tests in cases when we can’t avoid that each test action updates the system state? How do we cope with non-repeatable tests?
Answer: I have yet to find a case where the “Functional Isolation” techniques that I described don’t suffice. Use the existing structures in the system to isolate test cases from one another.
Question: Great talk! How to you suggest to keep (concurrent) test cased isolated if faking the system time?
Answer: Thanks 😉 This is one of those cases where using one deployed version of the system doesn’t work. In the case of “Time-Travel” tests, then each test does need its own version of the System under test. So for each time travel test you have to incur the cost of deploying and starting the system – these tests aren’t cheap!
Question: What ære the biggest challenges of implementing executable specifications in a team
Answer: I think that the tech is relatively simple, the hard parts are changing the way that people think about tests and testing. Which parts of this are *most* difficult depends on the team. Some teams find it very hard to move responsibility for the tests to developers. Others find it difficult to translate, often over-complex, too-large, requirements into sensible user-stories that make it easy to map from story to executable specification.
Question: If you use such effort on building a nice DSL for the tests… Why doesn’t the actual system not just have such a nice API?
Answer: Good question, I think that good design pays, wherever you apply it. But however good your API design, I advise that you keep a layer of “insulation” between your test cases and the API. If your API is a wonderful exercise in clarity and brevity, the map to domain language will be simple, but you still need a separate place to allow you to manage changes. Executable Specifications/Acceptance Tests are a special case. When writing them you will be expressing ideas at a different level of abstraction to what is needed through a programatic interface. So you want enough “wiggle-room” to allow you to cope with those variances.
Question: If I am a developer of System B, which is downstream from System A, I should write tests for the output of System A, to check if it still respects the interface. But, how do I know what are the inputs to the System A to make it output what I am expecting in my test?
Answer: That is a problem, but it is a smaller problem than doing ALL of your testing via system “A” which is what I am advising against. Let’s invert this question, if you are a developer of System “B”, how much do you care about up-stream system “A”? Write tests to exercise the system to the degree that you care about it. (Good design would suggest that you should care about it to the minimal degree). System “A” talks to my system, system “B”, so I can either confirm how it talks to my system with these tests, or I can cross my fingers and hope that I don’t get a call at 3am when system “A” decides to do something else 😉
Question: Test infrastructure is also code, how do you test the test infrastructure itself?
Answer: There is clearly a law of diminishing returns at play here. You can’t apply TDD for every test case. I aim to make my DSL clear and simple, high-level, enough that it doesn’t need testing (at the level of individual test-cases). Sometimes though, I will use TDD to develop widely-used, more complex, bits of my test infrastructure. I see automated testing in general and TDD in particular as a really important tool in a developers kit-bag. It is like having power tools. Sometimes, I may need to do something that is too simple for the power tools (using a regular screw-driver to change the batteries in my smoke alarm). Other times I will use the power tools because they will be faster and more reliable when I am doing something more complex (assembling a kitchen-cabinet) 😉
Question: Loosing the “checks and balances” aspect does not seam like a good idea. If it’s the developer owning the acceptance tests, won’t he just test what he thinks is valid?
Answer: I am afraid that I don’t buy the “Checks and balances” argument. IMO automated testing is less about testing and much more importantly about development process and quality in design. Automated testing makes me design and architect systems in a way that leads to better systems. It encourages modularity, separation of concerns and many other good properties this is true of both TDD and Acceptance Testing (ATTD). I think that professional testers add to the quality of testing, but they do this by educating development teams to do better, and by exploratory testing – not through taking ownership of Quality/Testing. Demming said “You can’t inspect quality in to a product”. Quality is designed in, testing is most important to the degree that it informs design decisions, and so it needs to be up-front and intimately involved in the development process.
Question: Assuming that its cost intensive, How much acceptance testing is enough?

Answer: You do spend a lot of time, and money on infrastructure, to adopt my recommended approach to testing. However, ALL of the data from the industry says that it pays for itself. This is a way of going faster with higher quality. If it wasn’t, I wouldn’t recommend it! What happens is that you trade-off the effort of building and maintaining automated tests against the effort of fixing bugs from production. Organisations that practice Continuous Delivery normal report at least an order of magnitude reduction in bugs in production. Imagine what you could do if you had 1 in 10 of the bugs that you currently have. Imagine if your team could spend 44% more time on new work?

Thanks to everyone for all the questions, enjoy the rest of the conference!
Posted in Acceptance Testing, Agile Development, Continuous Delivery, TDD | Leave a comment

Confessions of someone who should know better

Ever had that sickening feeling that you have lost some important data?

I have been travelling a lot lately and arrived home to find that my personal blog site, this site, was down.

I host this site via a hosting service, running WordPress, as well as a few other things.

This site started out as a personal thing, and I didn’t expect many readers. Turns out that I have more readers than I expected – Thank you!

However, somehow I never moved the site, in my mind, from “personal and of little consequence”, to “worth maintaining properly”. So my backups weren’t really up to date and while I have copies of all of the posts that I have made, and some that I decided not to post for one reason or another, I kept those as txt files, not a nice simple backup of my site 🙁

As I started digging into what was wrong, I got that sickening feeling! The DB on my host that contained the content of my blog site had vanished completely. I spent a few days trying to find what had happened and find the data for the posts, all to no avail.

So, after some code-archeology, I have managed to patch something together from the most recent backup, a shamefully old one, and the manual copies of my posts.

As a result I have certainly broken links to some of my more recent posts – very sorry.

I have also probably spammed anyone with an RSS feed for my site – very sorry again!

My site is back up and working. I think that it is close to what was there before, with one or two posts added, while I was at it, and possibly one or two omitted by accident (let me know if you see something missing that should be there)

I am sorry for any inconvenience that I have caused, hopefully normal service will now be resumed.

Plus you can bet that I will be more diligent in maintaining my backups in future :-/

Posted in Blog Housekeeping | 2 Comments