A Few Thoughts on Feature Flags

Posted on April 9, 2018 by davef

I confess that “Feature Flags” make me a bit nervous. Despite this I think them a useful and important tool in our ability to achieve Continuous Integration.

So why do they make me nervous? Well, they are a form of “branching” they are designed to isolate change and as I have described previously, I think that branching works against Continuous Integration.

There is a big difference between Feature Flags and VCS-based branches though. Feature Flags isolate at the level of behaviour rather than at the level of code. This is an important and valuable distinction.

“Feature Flags isolate at the level of behaviour rather than at the level of code”

One of my practical objections to the use of VCS branches for normal development is that they place a barrier to promiscuous refactoring. All of the best code-bases that I have worked in had a high churn-rate. We would change them often and make them better in small ways all the time. Branches tend to prevent us from doing that, Feature Flags, on the other hand, allow it.

The danger with Feature Flags is that they can introduce considerable complexity. Which version of your code do your test? Feature on or off? Both?

If “both”? you are on a journey into exponential complexity growth as you add more flags – Flag “A” on Flag “B” on, Flag “A” on Flag “B” off, and so on! This is a never-ending game that you can’t win in any definitive way.

I tend to employ a hierarchy of approaches to allow me to make progressive changes in my code running under Continuous Integration.

First, I prefer to release change directly, make a change and have people use it.

Next, I will use “dark-release” or “branch-by-abstraction”. Dark-release allows me to build up, and test, stuff that people aren’t using yet. Branch-by-abstraction encourages me to create abstractions in my code. These abstractions allow me to switch the implementation of these abstract features easily. It also fits with my style of coding where I care very much about separation-of-concerns and abstraction. Branch-by-abstraction can even allow me to run the old and new versions of a feature in parallel! This opens another world of possibilities for measuring the merits, or otherwise, of new features.

Only if none of these work will I use Feature Flags. This is largely because of the testing problem. For the types of systems that I have worked on for the past few years, I want to test what is running in production.

Another facet of this is that, on the whole, I prefer to make my Deployment Pipeline so efficient, that if I want to change the config of my system, even its Feature Flags, I will push the change through the pipeline, and so I can test it before release!

Posted in Agile Development, Continuous Integration, Effective Practices | 8 Comments

Continuous Integration and Feature Branching

Posted on March 30, 2018 by davef

Recently I spoke at the Pipeline Conference in London. I gave a talk on “Optimising Continuous Delivery” to a bunch of people who were self-selected as interested in Continuous Delivery, most of them would already consider themselves CD practitioners, Pipeline is a conference dedicated to learning about CD!

Early in my talk I described some of the ground-rules for CD, those practices that I consider “table-stakes” for playing. One of my slides is intentionally slightly jokey. It describes my advice for branching strategies in the context of CD.

dontBranch

I knew that this would be contentious, it is always contentious. This is the practice that I advise that I get most push-back on. EVERY TIME!

Before I had got home from the conference my twitter account ‘@davefarley77’ had gone a bit mad. Lots and lots of posts, for and against, questions and challenges and while a week later it has slowed down a bit, the rumblings continue.

I wrote about feature branching some years ago. I was about to say that the debate has moved on, but in reality I don’t think that it has. The same questions and issues arise. So this blog post is meant to add a few more thoughts on the topic.

The push-back that I get when I express my view that any form of branching is counter, in principle, to the ideas of Continuous Integration is varied.

At one extreme I get “Heretic, burn him at the stake” kind of feedback, at the other “Yes, but it can’t possibly work without Feature Branching – you must work in small teams and/or on trivially simple projects”.

The first point is fair enough, I am, constitutionally, a heretic. I like to be skeptical about “received wisdom” and question it.

In this case though, my views on branching are from experience rather than mere academic skepticism. I have been a professional software developer for nearly four decades now. I have tried most things over the years. I have refined my understanding of what works and what doesn’t on a lot of projects, trying a lot of different tools, technologies, methodologies and techniques.

In response to the second class of “push-back” I do sometimes work in small teams, but also with some of the biggest companies in the world. For the past three decades I think that it is fair to categorise most of my development work as at the more complex end of the scale. Which is one of the reasons that I take some of these disciplines quite so seriously.

I am an adherent of agile principles and take them to their extreme with my flavour of Continuous Delivery when I am in a position to decide, or influence the decision.

I first practiced a version of Continuous Integration in 1991. We had a continual rolling build, a home built version control system, written in shell-script, and even a few simple “unit tests” on our C++ project. This pre-dated the popularity of CI by a considerable margin, but it worked really well!

What I learned on this project, and on many others, small and MASSIVE, is that what really matters is feedback! Fast and high-quality. The longer that you defer feedback, the greater the risk that something unexpected, and usually bad, will happen.

This is one of the ideas that inspired the evolution from Continuous Integration to Continuous Delivery. We wanted better feedback, greater insight, into the effect of our changes, whatever their nature.

So you can tell, I am a believer in, and advocate for, Continuous Integration. We create better code when we get fast feedback on our changes all of the time.

CI is a publication based approach to development. It allows me to publish my ideas to the rest of my team and see the impact of them on others. It also alows the rest of my team to see, as it is evolving, the direction of my thinking. When teams practice CI what they get is the opportunity to “Fail Fast”. If something is a problem, they will spot it REALLY quickly, usually within a handful of minutes.

CI works best when publications/commits are frequent. We CI practitioners actively encourage commits multiple times per day. When I am working well, I am usually committing every 15 minutes or so. I practice TDD and so “Red-Green-Refactor-Commit” is my mantra.

This frequency doesn’t change with the complexity of the code or size of the team. It may change with how clearly I am thinking about the problem or with the maturity of the team and their level of commitment to CI.

What I mean by that, is that once bitten by the feedback bug, you will work VERY hard to feed your habit. If your build is too slow, work to speed it up. If your tests are too slow, write better tests. If your hardware is too slow on your build machines, buy bigger boxes! I have worked on teams on some very large codebases, with complex technologies that still managed to get the fast feedback that we needed to do high-quality work!

If you care enough, if you think this important enough, you can get feedback fast enough, whatever your scale! It is not always easy, but it has always been possible in every case that I have seen so far – including some difficult, challenging tech and some VERY large builds and test suites!

“What has all of this got to do with branching?” I hear you ask. Well if CI is about exposing our changes as frequently as possible, so that we can get great feedback on our ideas, branching, any form of branching, is about isolating change. A branch is, by-design, intended to hide change in one part of the code from other developers. It is antithetical to CI, the clue is in the name “CONTINUOUS INTEGRATION”!

To some degree this isolation may not matter too much. If you branch, but your branch is VERY short-lived, you may be able to get the benefits of CI. There are a couple of problems with this though. First, that this is not what most teams do. Most teams don’t merge their branch until the “feature” that they are working on is complete. This is called “Feature Branching”.

Feature Branching is very nice from the perspective of an individual developer, but sub-optimal from the perspective of a team. We would all like to be able to ignore what everyone else is doing and get on with our work. Unfortunately code isn’t like that. Even in very well factored code-bases with beautiful separation-of-concerns and wonderfully loosely-coupled components, some changes affect other parts of the system.

I am not naive enough to assert that Feature Branching can never work, you can make anything work if you try hard and are lucky. Even waterfall projects occasionally produced some software! My assertion is that feature branching is higher-risk and, at the limit, a less efficient approach.

whereToRunCI

The diagram above shows several paths from idea to working software in production. So if we want effective, high-quality feedback where in this diagram should we evaluate our changes? Point 1 is clearly no good, the changes on the branches, 5, 6 and 7, are never evaluated.

We could evaluate the changes after every merge to trunk, 2, 3 and 4. This is what lots of Feature branching teams do. The problem now is twofold:

1) We get no feedback on the quality of our work until we think that we are finished – Too Late!
2) We have zero visibility of what is happening on other branches and so our work may not merge. – Too Risky!

Before the HP Laserjet Firmware team made their move to Continuous Delivery, their global development team spent 5 times as much effort on merging changes between branches as on developing new features!

(See from time 47:16 in this presentation  also “A Practical Approach To Large Scale Agile Development”)

At this point my branch-obsessed interlocutors say “Yes, but merging is nearly free with modern tools”.

It is true! Modern distributed Version Control Systems, like GIT, have very good merge tools. They can only go so far though. Modern merge tools are good at the optimistic lock strategy of deferring locking things down until you see a conflict, at which point they request some help, your help. Most of the time merges are simple and automatic, but often enough, they are not.

As soon as you need to intervene in a merge there is a cost and until the point of merging you don’t know how big that cost will be. Ever got to merge some change that you have worked on for several days or a week, only to find that the differences are so great that you can’t merge? Lots of teams do find themselves in this position from time to time.

Back to our diagram. What feature branch teams sometimes do is run a dual CI system, they run CI on the branches AND after the merge to Trunk. This is certainly safer, but it is also slow.

As ever, the definitive point is the testing that happens at the point of merge to Trunk. It is only at this point that you can honestly say “Yes, my change works with everyone else’s.”. Before that, you are hoping that someone else hasn’t done something horrid on another branch that breaks your stuff when you merge.

This approach is safer because you are getting some feedback sooner, from the CI running on your feature branch, but this branch is telling lies. It is not the real story. This is not a change set that will ever make it into production, it isn’t integrated with other branches yet. So even if all your tests pass on this branch, some may fail when you merge. It is slow because you are now building and running everything at least twice for a given commit.

The real, authoritative feedback happens when you evaluate the set of changes, post merge, that will be deployed into production, until your branch is finished and merged onto Trunk, everything else is a guess.

CI advocates advise working on Trunk all the time. If you want to be pedantic, then yes, your local copy of the code is a form of branch, but what we mean by “all the time” is that we are going to make changes in tiny steps. Each change is itself atomic and leaves the code in a working state, meaning that the code continues to work and deliver value. We will usually commit many of these atomic changes every day. This often means that we are happy to deploy changes into production that are not yet complete, but don’t break anything!

CI, like every other engineering practice, comes with some compromises. It means that we are only allowed to commit changes that keep the system working. We NEVER intentionally commit a change that we know doesn’t work. If we do break something the build stops and rejects our change, that is the heart of CI.

This means that we have to grow our features over multiple commits, if we want regular, fast, authoritative feedback. This, in turn, changes the way that we go about designing our features. It feels more like we “grow” them through a sequence of commits rather than take them aside, design them and build them in isolation and then merge them.

This is a pretty big difference. I think that this is one of the reasons for the second category of push-backs that I tend to get from people who are more used to using branches.

Q: “Yes, but how do you make anything complex in 15 minutes?” 

A: You don’t, you break complex things into a series of small, simple changes.

Q: “But how can a team fix bugs in production?”

A: They feed the fixes in to the stream of commits, like any other change to the system.

Q: “Ah yes, but how do you do code reviews?”

A: Pair Programming is my preferred approach. You get better code reviews and much more.

Q: “Ah, but you can’t do this for software like XXX or using technology like YYY”

A: I have build systems-software, messaging systems, clustering systems, large volume data-base backed systems, whole enterprise systems, some of the highest performing trading software in the world, as well as web-sites, games and pretty much any other type of software that you can think of using this approach.

I have done it in Java, C#, C++, Python, Ruby, Javascript, shell-script, FPGA systems, Embedded software and COBOL. I have seen other teams using this approach on an even wider variety of technologies and products. I think it works!

CI is about speed and clarity of feedback. We want a definitive picture of the quality of our work, which means that we must evaluate, precisely, the code that will go into production. Anything else is guessing. We want our feedback fast and so we will optimise for that. We work to position the machinery that provides that feedback so that it can try our changes destined for production as soon as possible, that is, as close to the point that we made the changes as we can achieve.

Finding our that my code is stupid or broken within 2 minutes of typing it is very different to having to wait, even as short-a-time as an hour for that insight. It changes the way that I work. I can proceed faster, with more confidence and, when I do mess-up, I can step back with very little cost.

So we want definitive feedback fast. That means that anything that hides change gets in the way and slows us down. Any form of branching is antithetical to Continuous Integration.

If your branch lasts less than a day, my argument against it is weakened, but in that case I will pose the question “why bother with branches?”.

I work on Trunk, “master” in my GIT repos. I commit to master locally and push immediately, when I am networked, to my central master repo where CI runs. That’s it!

I do compromise the way that I work to achieve this. I use branch by abstraction, dark-releasing and sometimes feature-flags. What I get in return is fast, definitive (at least to the quality of my testing) feedback.

Last years “State of DevOps Report” claimed that my style of development is a defining characteristic of “High Performing Teams”. If you are not merging your changes to Trunk at least daily, it predicts that your outcomes are more closely aligned with “Lower Performing Teams”.

There is a lot more to this, CI is not a naive approach it is well-thought out and very widely practiced in some of the most successful companies in the world. Trunk-based development is a core practice to CI and CD, it really is very difficult to achieve all of the benefits of CI or CD in the absence of Trunk-based development. You can read more about these ideas on the excellent Trunk-Based-Development site.

Posted in Agile Development, Continuous Delivery, Continuous Integration, Effective Practices, Feature Branching | 81 Comments

Answers to GOTO Cph 2017 Questions

Posted on October 2, 2017 by davef

I gave a presentation on my recommended approach to Acceptance Testing today, here at GOTO Copenhagen.

You can see an earlier version of this talk, from another conference here. GOTO will be publishing their version soon 😉

I ran out of time for questions, but here are my answers to questions submitted via the GOTO app…

Question: Those slides could use a designers touch, though 🙂
Answer: Fair enough, I am not a designer 😉
Question: How do we obtain repeatable tests in cases when we can’t avoid that each test action updates the system state? How do we cope with non-repeatable tests?
Answer: I have yet to find a case where the “Functional Isolation” techniques that I described don’t suffice. Use the existing structures in the system to isolate test cases from one another.
Question: Great talk! How to you suggest to keep (concurrent) test cased isolated if faking the system time?
Answer: Thanks 😉 This is one of those cases where using one deployed version of the system doesn’t work. In the case of “Time-Travel” tests, then each test does need its own version of the System under test. So for each time travel test you have to incur the cost of deploying and starting the system – these tests aren’t cheap!
Question: What ære the biggest challenges of implementing executable specifications in a team
Answer: I think that the tech is relatively simple, the hard parts are changing the way that people think about tests and testing. Which parts of this are *most* difficult depends on the team. Some teams find it very hard to move responsibility for the tests to developers. Others find it difficult to translate, often over-complex, too-large, requirements into sensible user-stories that make it easy to map from story to executable specification.
Question: If you use such effort on building a nice DSL for the tests… Why doesn’t the actual system not just have such a nice API?
Answer: Good question, I think that good design pays, wherever you apply it. But however good your API design, I advise that you keep a layer of “insulation” between your test cases and the API. If your API is a wonderful exercise in clarity and brevity, the map to domain language will be simple, but you still need a separate place to allow you to manage changes. Executable Specifications/Acceptance Tests are a special case. When writing them you will be expressing ideas at a different level of abstraction to what is needed through a programatic interface. So you want enough “wiggle-room” to allow you to cope with those variances.
Question: If I am a developer of System B, which is downstream from System A, I should write tests for the output of System A, to check if it still respects the interface. But, how do I know what are the inputs to the System A to make it output what I am expecting in my test?
Answer: That is a problem, but it is a smaller problem than doing ALL of your testing via system “A” which is what I am advising against. Let’s invert this question, if you are a developer of System “B”, how much do you care about up-stream system “A”? Write tests to exercise the system to the degree that you care about it. (Good design would suggest that you should care about it to the minimal degree). System “A” talks to my system, system “B”, so I can either confirm how it talks to my system with these tests, or I can cross my fingers and hope that I don’t get a call at 3am when system “A” decides to do something else 😉
Question: Test infrastructure is also code, how do you test the test infrastructure itself?
Answer: There is clearly a law of diminishing returns at play here. You can’t apply TDD for every test case. I aim to make my DSL clear and simple, high-level, enough that it doesn’t need testing (at the level of individual test-cases). Sometimes though, I will use TDD to develop widely-used, more complex, bits of my test infrastructure. I see automated testing in general and TDD in particular as a really important tool in a developers kit-bag. It is like having power tools. Sometimes, I may need to do something that is too simple for the power tools (using a regular screw-driver to change the batteries in my smoke alarm). Other times I will use the power tools because they will be faster and more reliable when I am doing something more complex (assembling a kitchen-cabinet) 😉
Question: Loosing the “checks and balances” aspect does not seam like a good idea. If it’s the developer owning the acceptance tests, won’t he just test what he thinks is valid?
Answer: I am afraid that I don’t buy the “Checks and balances” argument. IMO automated testing is less about testing and much more importantly about development process and quality in design. Automated testing makes me design and architect systems in a way that leads to better systems. It encourages modularity, separation of concerns and many other good properties this is true of both TDD and Acceptance Testing (ATTD). I think that professional testers add to the quality of testing, but they do this by educating development teams to do better, and by exploratory testing – not through taking ownership of Quality/Testing. Demming said “You can’t inspect quality in to a product”. Quality is designed in, testing is most important to the degree that it informs design decisions, and so it needs to be up-front and intimately involved in the development process.
Question: Assuming that its cost intensive, How much acceptance testing is enough?

Answer: You do spend a lot of time, and money on infrastructure, to adopt my recommended approach to testing. However, ALL of the data from the industry says that it pays for itself. This is a way of going faster with higher quality. If it wasn’t, I wouldn’t recommend it! What happens is that you trade-off the effort of building and maintaining automated tests against the effort of fixing bugs from production. Organisations that practice Continuous Delivery normal report at least an order of magnitude reduction in bugs in production. Imagine what you could do if you had 1 in 10 of the bugs that you currently have. Imagine if your team could spend 44% more time on new work?

Thanks to everyone for all the questions, enjoy the rest of the conference!
Posted in Acceptance Testing, Agile Development, Continuous Delivery, TDD | Leave a comment

Confessions of someone who should know better

Posted on January 24, 2017 by davef

Ever had that sickening feeling that you have lost some important data?

I have been travelling a lot lately and arrived home to find that my personal blog site, this site, was down.

I host this site via a hosting service, running WordPress, as well as a few other things.

This site started out as a personal thing, and I didn’t expect many readers. Turns out that I have more readers than I expected – Thank you!

However, somehow I never moved the site, in my mind, from “personal and of little consequence”, to “worth maintaining properly”. So my backups weren’t really up to date and while I have copies of all of the posts that I have made, and some that I decided not to post for one reason or another, I kept those as txt files, not a nice simple backup of my site 🙁

As I started digging into what was wrong, I got that sickening feeling! The DB on my host that contained the content of my blog site had vanished completely. I spent a few days trying to find what had happened and find the data for the posts, all to no avail.

So, after some code-archeology, I have managed to patch something together from the most recent backup, a shamefully old one, and the manual copies of my posts.

As a result I have certainly broken links to some of my more recent posts – very sorry.

I have also probably spammed anyone with an RSS feed for my site – very sorry again!

My site is back up and working. I think that it is close to what was there before, with one or two posts added, while I was at it, and possibly one or two omitted by accident (let me know if you see something missing that should be there)

I am sorry for any inconvenience that I have caused, hopefully normal service will now be resumed.

Plus you can bet that I will be more diligent in maintaining my backups in future :-/

Posted in Blog Housekeeping | 2 Comments

Diversity

Posted on January 24, 2017 by davef

I inadvertently found myself in the middle of a minor Twitter storm on the topic of diversity. The organisers of a conference that I attended made some, to me, intemperate remarks on the subject.

They were asked why there were so few women on the programme and responded in a way that came across as being, again, to my mind, overly aggressive and defensive.

I am a big fan of Twitter, but it is not really an effective vehicle through which to explore complex ideas. You can’t represent nuance in 140 characters.

So here are my thoughts on the topic.

I believe that our industry has a shameful history in terms of diversity. It is largely populated by young, white guys, at least in Western Europe and the USA. This is disproportionately the case, even compared to other technical disciplines.

This matters for a variety of reasons, social, emotional, political but also pragmatic.

It seems obvious to me that if I recruit people who are passionate about programming, enjoy science fiction, are obsessed with aeroplanes, aerobatics and physics, like playing the guitar and driving too fast (people like me) then we will tend to jump to similar conclusions and make similar mistakes.

I like working with people that have different ideas to me. I know that when I do this, it brings that best out of me, and I think it brings the best out of them. I think that we are at out most creative when we value ideas and work in teams that trust one-another sufficiently to feel free to debate those ideas freely and vigorously.

Software development is a VERY difficult thing to do well. I think we should maximise our chances of success by doing whatever it takes to be intelligent, creative and do great work. I believe that part of that is creating diverse teams. Not just having a smattering of women around, but teams populated with people from all sorts of different backgrounds, education, ethnic groups, sex or sexual orientation. I believe that this is one of the hallmarks of truly great teams.

The problem is that we live in an unfair world. I don’t know enough to solve the problems of inequality in our world. I am a software developer. I care deeply about these problems and believe that they represent an injustice, and a loss of potentialy great contributions. I believe that we can do better.

However, I also think that this is not a problem of the software conference industry. Poor representation of women at conferences did not get us into the problematic position that we find ourselves in. I have some sympathies for the conference organisers. Even though they responded in a manner that I thought at best intemperate, at worst inappropriate, there were mitigating circumstances.

At this conference there were 25 speakers. Of those, 2 were women. The organisers were criticised for the lack of women speakers, but actually in terms of representation of women in our industry that does not feel too far off being a proportionate number. The organisers words came across badly, but the English is not the first language of these conference organisers and I think that a reasonable interpretation of what they said is that they did not really get the nuance of what they implied with their comments.

I suppose that the key question is, what can be done to change the situation that we, as an industry, find ourselves in?

Well there are several positive examples that I am aware of. Even if we constrain ourselves to the conference arena. The Pipeline Conference, in London, does an excellent job of actively encouraging female speakers. They also work to eliminate subconscious bias in the selection process for talks by using a blind evaluation process. Submissions are stripped of identifiers so that the selection committee can’t be swayed by the sex, ethnic background, or fame of the prospective speakers.

QCon work hard to encourage and promote female speakers for their events and operate a code-of conduct for speakers and other participants that encourages and open and respectful approach to all.

These are important things, conferences are one public face of our industry and the roster of speakers will provide, albeit subliminally, an impression of what a software professional looks like and how they act.

However, even if we had a 50/50 split of men and women and a perfect representative sample of all cultural or ethnic groups at every conference, it would mean nothing if we don’t address the real problem, which is that there aren’t enough of these people in our industry. Worse than that, focusing only on women for a moment, they are after all 50% of the population, our industry no longer appeals to women and girls. We have driven them away to the extent that few even consider software development as a career.

I think that software is an important thing for the world. I feel privileged to have found a career that I love, that also happens to be interesting, challenging and pays pretty well. I don’t want that privilege to belong only to people like me.

I have done what I could in my career to treat people with respect, whatever their sex, sexual orientation, ethnic group or religious persuasion. That is not enough.

Posted in Culture | 1 Comment

Mob Rule?

Posted on October 7, 2016 by davef

I was at a conference last year where I saw Woody Zuill talking about “Mob Programming”. You can see that talk here

A very simple description of Mob programming, for those of you who don’t have time to watch Woody’s presentation, is “All the team working together on the same stuff”. Instead of dividing work up and allocating it to individuals or pairs, the whole team sits together and advises a “Driver” who has the only keyboard.

Interested but Skeptical

I thought it was an interesting, thought-provoking, even challenging idea. I confess that my first reaction was skepticism. To be honest, as a grumpy old man in my 50s, with a scientific-rationalist world-view my reaction to most things is skepticism. I think of it as a healthy first response 😉

I thought about my skeptical responses, “That can’t possibly be as efficient as the team dividing the work”, “Surely some people will just sit back and not really contribute”, “It is going to be dominated by a few forceful individuals”, “How could you design anything complex?”. Once I started voicing them, they seemed familiar. These are precisely the kind of responses that I get when I talk to teams about adopting Pair Programming.

Now, I am a passionate advocate for Pair Programming. I believe that it makes teams stronger, more efficient, more cohesive and allows them to produce significantly higher-quality work. So I was in a quandary.

On one hand, “this can’t possibly be efficient, can it?” on the other “Working collaboratively produces better results”. I am scientific rationalist enough to retain an open-mind. I assumed that that would be it. I assumed that, given the nature of my work as a consultant, I would never experience Mob Programming personally and so would only ever see it from the perspective of a distant, outside observer. I was wrong.

Invited to join the Mob

I have some friends working in a start-up company, http://www.navetas.com/ who have recently experimented with Mob Programming and have been practicing it for a few months now. I know them through my good friend, Dave Hounslow, who used to work there. The team very kindly invited both of us to spend the day with them and “join the mob”.

Before trying Mob Programming, the team was already fairly advanced in their use of agile development and Continuous Delivery. Dave had helped them to establish a strong culture of automated testing and an effective deployment pipeline. They were used to working collaboratively and doing pair programming, TDD and automated acceptance testing.

The team is fairly small, 5 developers, they saw a presentation on Mob programming and decided to try it as an experiment for a single iteration, and have never gone back to their previous mode of work, pair-programming.

Introductions, Process and People

Dave and I arrived and after coffee and introductions, we attended the team stand-up meeting. The team are thinking of changing the stand-up because it no longer has a role of establishing a shared understanding, they do that all day, every day, working together in the Mob. However there is one team member who works remotely on customer service and so this is an opportunity to catch-up with her.

We then spent a bit of time in front of a whiteboard while the team described their system to us. Dave knew it a bit from when he used to work there, it was new to me. They described their architecture and then the story that we would be working on, and then we began.

The team have invested a bit of time and infrastructure to support their mob programming. They have a couple of nice big monitors so that everyone can see what is going on as the code evolves and a timer that reminds them, when to swap who is driving.

The approach works by giving everyone a turn at the keyboard, including newbies like Dave and I. The timer is set for 15 minutes. Each person gets 15 minutes at the keyboard and then everyone shifts where they are sitting with a new person taking the keyboard. It is not totally rigid, so if you are at the keyboard and in the middle of typing a word, you can finish, but the team try to respect the time-slots. They avoid one person hogging the keyboard.

We sat in a loose semi-circle with the person at the keyboard placed at the centre. We all then offered advice on what to do and where to go next. When the time was up everyone shuffled round one place, musical chairs without the music. The next person in sequence would take the keyboard and we would continue the conversation and design of the code. I think that the simple act of getting up and shifting seats helped keep the engagement going throughout the day. Tiny as it was, the act of moving sharpened the focus, just a little bit.

Mob Programming1

Sounds chaotic doesn’t it? However, it wasn’t. There were occasions when the discussion went on enough that there was no typing during the 15 minute period but on the whole that wasn’t the case. On the whole we made steady progress towards the aims of the story.

Mob Programming2

There was a lot of talking, this wasn’t dominated by one or two people everyone had their say. There was, inevitably some variance. The level of experience of the team varied widely both in terms of general software development experience and in terms of experience of this project. So at different points different people had more or less to contribute. I was a complete newby to the system and so I asked more questions than the others and could mostly contribute on general issues of design and coding rather than specifics of the existing system. Others knew the system very well and so would contribute more when specific questions arose. This is one of the benefits!

Optimise for Thinking, Not Typing

The story we were working on was one of those stories that was exploring some new ideas in the system. It was gently challenging existing architectural assumptions. At least that was the way that I perceived it. This was a good thing. This wasn’t a trivial change, we had new territory to explore. We certainly spent more time talking than typing, but then I have never admired “lines of code” as a useful metric for software. I much prefer to optimise for thinking rather than typing. So while the amount of code that we produced during the day was relatively small, I think that it was higher quality than any of us would have produced alone.

The conversations were wide ranging, and often I felt that the presence of two outsiders, Dave and me, tilted the conversation more in the direction of reviewing some past decision than may otherwise have been the case. However, I also felt as though we both added some value to the discussions and the final output of the day, working code.

Assumptions Fulfilled?

So what about the questions that I started with?

“It can’t possibly be as efficient…”. Well the team have tracked their velocity and have seen a significant increase in the number of stories that they produce. I know that velocity is really only a pseudo-measure of progress, who is to say that this weeks stories are the same size as last? Nevertheless, the team subjectively feel that they are now moving significantly more quickly and producing a higher-quality output and the data that they have seems to back this up.

“People will sit back and not contribute”. There were some people that spoke less than others, but everyone was engaged throughout the day and everyone contributed to some degree.

“It will be dominated by forceful individuals”. There were a few “forceful personalities” present, myself and Dave included. Nevertheless the team seemed to me to listen carefully and consider ideas from everyone. It is certainly true that some of us present talked more and were a bit more directive than others in the group, nevertheless I felt as though the outcome was a group-driven thing.

“How could you design anything complex?”. This is an old chestnut that people new to TDD and Pair Programming raise all the time. I am completely relaxed about the ideas of incremental design. In fact I believe that it is the only way that we can solve complex problems with high quality. I was pleased that the story that was chosen for the day had a bit of substance to it. It wasn’t a deeply challenging problem, but did stress some architectural assumptions. I am confident that we, the whole group, moved the thinking about the architecture of the system forwards during the course of our work that day.

Justified Skepticism?

So what do I think about Mob Programming now that I have tried it? Well I am still on the fence. I finished that day having had my beliefs challenged. This is not a crazy, inappropriate waste of time by any stretch of the imagination.

This was a small team that was considerably disrupted by having two newbys, Dave and me, drop-in for the day. I think tha for any team this small, this would have been disruptive. I seriously doubt that, for most teams, the new people would have made much of a contribution. Dave and I are both experienced software developers and both used to working as consultants and adding value quickly. Nevertheless I think that we added more value, more quickly than would usually be the case. We made suggestions that the team tried immediately. We learned about the system more quickly, to a level of detail that surprised me. Again, in any other process, I think that we would have spent more time either doing introductory stuff or applied a narrower focus and worked in a smaller area of the code – on our first day.

I was extremely impressed that this small team could accommodate the disruption of, nearly, doubling in size for a day and not only do useful work, but actually do so in a way that moved on their thinking.

At the human level this is a nice way to work. You have the conversations that are important. You share the context of every decision with the whole team all the time. You laugh and joke and share the successes as well as the disappointments.

These days I work as an independent consultant, advising people on how to create better software faster. I am intrigued at the prospect of using Mob programming as a mechanism to introduce small teams to new ideas. I think it could be a wonderful instructional/learning tool. If anyone fancies carrying out that experiment you should give me a call 😉

So where are my reservations, why do I feel like I am still “on the fence” rather than a true believer? Well, there was a bit coaching by one of the participants to organise the others. I honestly don’t know if the process would have worked better or worse without him directing it quite so strongly, but I can imagine it breaking down if you had the wrong mix of people. That is a weak criticism, any process or approach can be disrupted by the wrong person or group of people. My point is that if you have a bad pair, you can move on and pair with someone else the next day. If you have a bad Mob you have to be fairly strong minded to face the problem and fix it. It will force you to have some difficult conversations. Perhaps no bad thing if you do it, but I know of many teams that would shy away from such a social challenge.

There must be some natural limit. The idea of a Mob of 200 people working on the same thing is ludicrous. So where is the boundary? I am a believer in the effectiveness of small teams. So I wouldn’t have a team of 200 people in the first place. However, I wonder how well this would work with larger, still small, teams? There were 6 of us in this Mob and it worked well. Would it have continued to do so with 8, 10 or 12? If I am honest I think that a team of 12 is too big anyway, but it is a valid question, what are the boundaries?

There are times when I want some time to think. If I am pairing and I hit such a point it is simple to say to my pair – “let’s take a break and come back to this in 30 minutes”. It is harder to make that call if everyone is working in a Mob.

On the whole I had a fascinating day. I would like to extend my thanks to the folks at Navetas for inviting me to join their Mob and experience this approach first hand. It was a lot of fun and I learned a lot.

Posted in Agile Development, Culture, Effective Practices | Leave a comment

Test *Driven* Development

Posted on June 29, 2016 by davef

Before Test Driven Development (TDD) the only thing that applied a pressure for high-quality in software development was the knowledge, experience and commitment of an individual software developer.

After TDD there was something else.

High quality in software is widely agreed to include the following properties.

High quality software:

    • Is modular.
    • Is loosely-coupled.
    • Has high cohesion.
    • Has a good separation of concerns.
    • Exhibits information hiding.

Test Driven Development is only partially about testing, of much greater importance is its impact on design.

Test Driven Development is development (design) driven by tests. In Test Driven Development we write the test before writing code to make the test pass. This is distinct from Unit Testing. TDD is much more than “Good unit testing”.

Writing the test first is important, it means that we always end up with “testable” code.

What makes code testable?

    • It is modular.
    • Loosely-coupled.
    • Has high cohesion.
    • Has a good separation of concerns.
    • Exhibits information hiding.

Precisely the same properties as those of high quality code. So with the introduction of TDD we now have something else, beyond the knowledge, experience and commitment of a programmer to push us in the direction of high-quality. Cool!

Like to know more?…

http://www.davefarley.net/?p=180

http://c2.com/cgi/wiki?TestDrivenDevelopment

http://www.agiledata.org/essays/tdd.html

http://butunclebob.com/ArticleS.UncleBob.TheThreeRulesOfTdd

http://unitmm.sourceforge.net/fibonacci_example.shtml

http://clean-cpp.org/test-driven-development/

http://agile2007.agilealliance.org/downloads/presentations/TDD-Cpp-Agile2007-HandsOnTddInCpp.ppt_801.pdf

http://www.growing-object-oriented-software.com/

Posted in Agile Development, Continuous Delivery, Effective Practices, Software Design, TDD | 4 Comments

Cycle-Time

Posted on June 29, 2016 by davef

Motivation is a slippery thing. My favourite example is described by the writer Dan Pink. He tells the true story of a Nursery who, like many Nurseries, had a problem with parents turning up late to collect their children. This is a big problem for such organisations. So what do you do in this situation? Well the obvious answer, which this Nursery tried, was to introduce a series of fines. If you arrived later than you should, you incurred a fine. If you arrived very late you incurred a bigger fine.

It is obvious that that is a deterrent, right? Well no. In fact the late collection problem got dramatically worse. What had happened is that previously the parents understood that there was a social compact. You collected your children on-time because it was bad manners, if nothing else, not to. Now the Nursery had put a price on it. The fine was the fee for looking after your children for longer. The Nursery had removed the social incentive for doing the right thing and replaced it with a financial one that described how much it cost to do the wrong thing. Parents decided to pay for the extra child-care!

These things are done with good intentions but the results are counter-productive. As I said, motivation is a slippery thing!

Metrics in particular, one form of motivation, are very difficult to get right. We often hear organisations talk about being “data-driven” and “the importance of KPIs” in driving good behaviours, but do these things really work? It is all too easy to define a metric that drives exactly the wrong behaviour.

How many times have you seen something that looks crazy and asked “why on Earth do we do that?” only to hear the answer “It is because that group of people are measured on xyz”.

“Why do our sales people sell stuff that isn’t ready?”, “Because they are incentivised to sell”.

“Why do our developers create poor quality code?”, “Because they are incentivised to create lots of features”.

“Why do our operations people slow things down?”, “Because they are incentivised on the stability of the system”.

However, there is one measure that, as far as I have seen so far, does not lead to any inappropriate gaming or misdirected incentives – Cycle Time!

I argue that Continuous Delivery is really about one thing. Having an idea, getting that idea into the hands of our users and figuring our what they make of it. So we should optimise our software development processes for that. Whatever it takes.

Cycle Time is a measure of that process. Imagine the simplest change to your production system that you can think of. We want it to be simple so that we can ignore the variable cost of development. Now imagine that change going through all of the normal processes to get it prioritised, scheduled, defined, implemented, tested, verified, documented and deployed into production. Every step that a change to production would normally take. The time that it takes to complete all of those steps, plus the time that the change spends waiting between steps, is your Cycle Time. This is a great proxy for measuring the time from “idea” to “valuable software in the hands of users”.

I believe that if you take an empirical, iterative approach to reducing Cycle-Time then pretty much all of Agile Development, Lean thinking, DevOps and Continuous Delivery practice will fall out as a natural consequence.

I once worked on a demanding, high-performance complex system that processes billions of dollars of other people’s money on a daily basis. This was a complex enterprise system, it included high-performance services, account management, public APIs, Web UIs, administration tools, multiple third-party integrations in a variety of different technologies, data-warehouses the lot. We had a Cycle Time of 57 minutes. In 57 minutes we could evaluate any change to our production system and, if all the tests passed, be in a position to release that change into the hands of users.

Now think about the consequences of being able to do that.

If you have a Cycle Time of 57 minutes, you can’t afford the communications overhead of large teams. You need small compact, cross-functional, efficient teams.

You can’t afford the hand-overs that are implicit in siloed teams. If you divide your development effort up into technical specialisms you will be too slow. You need cross-functional collaborative teams to ensure a continual flow of changes.

You can’t rely on manual regression testing. You need a great story on automated testing. Human beings are too slow, too inefficient, too error prone and too expensive.

You can’t rely on manual configuration and management of your test and production environments. You need to automate the configuration management, automate deployment and you will need a good story on “Infrastructure as code”.

You can’t have a Cycle Time of 57 minutes and have hand-offs between Dev and Ops.

You can’t have a Cycle Time of 57 minutes if your business can’t maintain a constant smooth flow of ideas.

You have to be very good at a lot of aspects of software development to achieve this kind of cycle-time. If you can confidently evaluate your changes to the point where you are happy to release into production in under an hour, without any further work, you are doing VERY well!

Optimising for short Cycle Time drives good behaviours. It is not just that you have to be good to achieve this, striving to improve your cycle-time will help you to improve your development process, culture and technology. It will force you to address impediments and inefficiencies that get in your way. This is a metric that doesn’t seem to have any bad side effects.

Many people are nervous that reducing Cycle Time will reduce quality. My experience, and that of the industry, is that the reverse is true. What happens is that by reducing Cycle Time you reduce batch-size. By reducing batch-size you reduce the risk of each change. Each change becomes simpler and lower risk. 66% of organisations that practice claim to practice Continuous Delivery say that quality goes up, not down1. Personally I am not too sure what the other 34% are doing wrong 😉

If you have a short Cycle-time, you can, and will, release change in small batches. Think about each change. Each change will be small and simple. Easy to reason about. If you release only once every few months, then you will be storing up lots of changes. Let’s imagine that each change has a small amount of risk associated with it. So the total risk for any release is going to be the sum of all of those risks.

Hmmm, not quite! As well as the sum of the risks associated with each change, there is going to be a combinatorial effect. What if my change interacts with your change? So there is an additional risk associated with the collection of changes. The probability of one of these risks being realised will grow exponentially as more changes are combined. The more changes are released together, the higher the risk that two, or more, changes will interact in unexpected ways. So the total risk is going to be something like the sum of all the risks associated with each change plus the risk that two or more changes will interact badly. Now imagine releasing changes one at a time, the second set of risks, the risks for which the probability of them occurring will increase exponentially with the number of changes, disappear all together. So overall many small changes is a much less risky strategy than fewer larger changes.

A few years ago I worked with a team building some complex software in C++. This development team was very good. They had adopted an automated testing approach some years before. They were well ahead of industry norms in that they operated a process based on an overnight build. Each night their automated systems would run, build the software and run their automated tests against it. The build and tests took about 9 hours to complete. Each morning the team would look at the results and there would be a significant number of test failures.

I spoke to one of the developers who had been working this way for the past three years. He told me that in that three year period there had been four occasions when all of the tests had passed.

So, the team did what teams do and adapted. Each morning they would look at the test results and only release the modules for which all of the the tests had passed. This is a reasonable strategy as long as none of the components interact with one another. Mostly they didn’t, but some components did. So now the team is releasing untested combinations of software into production which may or may not work together. As a result this team often saw problems deploying new features into production because of incompatibilities with older versions of components that they depended upon.

I argued that cycle-time was important, a driver for good behaviour and outcomes. I won the argument enough to give it a try.

We worked hard on the build. We invested a lot of time, money and effort on experimenting with different approaches. We parallelised the build, improved incrementalism, we bought some massive severs and triaged the tests into groups, dividing the build into a deployment pipeline. We moved from a 9 hour overnight build to a 12 minute commit stage (running the vast majority of the tests) followed by a slower (1 hour) Acceptance test stage. The “Acceptance Test” designation was fairly arbitrary in this case. If a test was too slow, we moved it to the “Acceptance Test Stage”.

The results were quite dramatic. In the first two week period, following the introduction of this new build, we saw three builds where all of the tests passed – compared to four in the previous three years. In the following two week period there were multiple successful (all tests passing) builds every day. The process now switched, instead of cherry-picking modules with passing tests, we could release all of the software together, or not at all. Each morning we would simply deploy the newest release candidate that had passed all the tests.

Now we could have more confidence that these components would work together. Now we could begin to improve our testing of scenarios that crossed the boundaries between components. Now we could be more thorough!

Reducing cycle-time drives good behaviours. It encourages us to establish concrete, efficient feedback loops that allow us to learn and adapt. The team in my war-story above was not different before and after the change in process. The change in approach, the focus on cycle-time, gave them insight into what was going wrong and an opportunity to learn. They could quickly and efficiently experiment with solutions to any problems that arose. This is a very powerful thing!

Cycle-time drives us in the direction of lower-risk release strategies. It encourages good practice and it moves us in the direction of higher-quality development practices. I encourage you to optimise your development process to reduce cycle-time. I believe that you will find that it improves almost everything that you do.

1 CA Technologies “DevOps Survey” 2015

Posted in Agile Development, Continuous Delivery, Culture, Effective Practices | 2 Comments

RedGate Webinar Q&A

Posted on May 11, 2016 by davef

I recently took part in a Webinar for Database tools vendor Redgate. At the end of the Webinar we ran out of time for some of the questions that people had submitted, so this blog post provides my answers to those questions.

If you would like to see the Webinar you can find it here.

Q&A

Q: “How can we overcome the “we’ve always done it that way” group-think mentality?”

Dave: For me the question is, “is your process working now as well as you want it to?” if not I think you should try something else.  

I believe that we have found a better way to deliver valuable, high-quality, software to the organisations that employ us. The trouble is that it is a very different way of working. Mostly people are very wary of change, particularly in software development, where we have promised a lot before and not delivered.  

The only way I know to move a “group-think” position is gradually. You need to make a positive difference and win trust. It is about looking at real problems and solving them, often one at a time.  

I believe that we, the software industry, are in a better place than we were, because we finally have the experience to know what works and what does not. The trick now is to migrate to the approaches that work. This take learning, because the new approaches are very different to the old, it challenges old assumptions. It is helpful to get some guidance, hire people that have some experience of this new way of working, read the literature, and carry out small, controlled experiments in areas of your process and business that will make a difference.  

I often recommend to my clients that they perform a “Value-stream analysis” to figure out where they are efficient at software delivery and where they are not. This is often an enlightening exercise, allowing them to easily spot points that can be improved. Sometimes this is technology, more often it is about getting the right people to communicate effectively. 

Once you have improved this problem, you will have improved the situation and gained a little “capital” in the form of trust that will allow you to challenge other sacred-cows. This is a slow process, but for a pre-existing organization it is the only way that I know. 

Q: “What advice would you have for gaining management buy-in for continuous delivery?”

Dave: Continuous Delivery is well-aligned with management ambitions. We optimise to delivery new ideas, in the form of working software, to our users as quickly and efficiently as possible. The data from companies that have adopted CD is compelling, it improves their efficiency and their bottom-line. Many of the most effective software companies in the world employ CD. 

The problem is not really the ideals, it is the practice, what it takes to get there. CD organizations look different to others. They tend to have many small teams instead of fewer large ones. Each team has a very high degree of autonomy, many don’t really have “management” in the traditional sense. So this can be very challenging to more traditional organizations. 

The good news is that the way to adopt CD is by incremental steps. Each of these steps is valuable in it’s own right, and so each can be seen as a positive step. If you don’t user version control – start. If you don’t work iteratively, and regularly reflect on the outcomes of your work so that you can correct and improve – start that. If you don’t employ test automation, or deployment automation or effective configuration management start those things too. Each of these steps will bring a different benefit, over time they reinforce one-another so you get more than the sum of the parts. 

There are several CD maturity models, there is one in the back of my book, which can offer guidance on what to try next here is another that I have used: http://www.infoq.com/articles/Continuous-Delivery-Maturity-Model 

Q: “We are very early in the stages of DB CD process changes, what are the most important issues to tackle early?” 

Dave: That is quite a tough question to answer without the consultants lament “It depends” 😉 

I think that the fundamental idea that underpins CD is to take an experimental approach to everything, technology, process, organization the lot. Try new things in small controlled steps so that if things go wrong you can learn from it rather than regret it. 

At a more technical level, I think that version controlling pretty much everything, automated testing and continuous integration are corner-stones. If you are starting from scratch, it is much easier to start well with automated testing and continuous integration than to add these later. It is not impossible to add them later, it is just more difficult. 

So be very strict with yourselves at first and try working so that you don’t make ANY change to a production system without some form of test. This will feel very hard at first if you are new to this, but it really is possible. 

Q: “Are there any best practices you’d especially recommend we bear in mind?” 

Dave: There is a lot to CD. I tend to take a very broad view of its scope and so it encompasses much of software development. At that level the best practices are grounded in Lean and Agile principles. Small, self-directed teams, working to very high quality standards, employing high levels of automation for tests and deployment are foundational. 

At the technical level there are lots at all different levels of granularity. I guess the key idea from my book is the idea of the “Deployment Pipeline” this is the idea of automating the route to production. A good mental model for this is to imagine that every change that is destined for production gives birth to a release-candidate. The job of the deployment pipeline is to prove that a release candidate is NOT FIT to make it into production. If a test fails, we throw the release candidate away. 

Q: “What are some common practical issues that people encounter during the implementation of CD?” 

Dave: I have covered some of this in the preceding answers. Most of the problems are people problems. It is hard to break old habits. At the technical end, the most common problems that I have seen have been very slow, inefficient builds, poor, or non-existent, automated deployment systems and poor, or non-existent, automated tests. 

Q: “What would be the fastest way to actually perform CD?” 

Dave: The simplest way to start is to to start from scratch, with a blank sheet. It is easier to start a new project or new company this way than to migrate an existing one.  

I think it helps to get help from people that have done this before. Hire people with these skills and learn from them. 

Q: “We deal with HIPPA regulated data and I am personally unsure of letting this data out. How does CD typically get implemented in highly regulated environments? Are there particular challenges?” 

Dave: The only challenge that I perceive is that regulators are often unfamiliar with the ideas and so their assumptions of what good regulatory conformance looks like is tailored with, what to me, looks like an outdated assumption of development practice. 

My experience of working in heavily regulated industries, mostly finance in different countries, is that the regulators quickly appreciate this stuff and they *love* it.  

CD gives almost ideal traceability, because of tour very rigorous approach to version control and the high-levels of automation that we employ we get FULL traceability of every change, almost as a side-effect. In the organizations where I have worked in the finance industry, we have been used as bench-marks for what good regulatory compliance looks like. 

So the challenge is educating your regulators, once they get it they will love it. 

Q: “How should a data warehouse deal with a source database which is in a CD pipeline?” 

Dave: As usual, it depends. The simplest approach is to treat it like any other part of the system and write tests to assert that changes work. Run these tests as part of your deployment pipeline.  

If not you need to take a more distributed, micro-service style approach. In this approach try and minimize the coupling between the Data Warehouse and the up-stream data sources. Provide well-defined, general, interfaces to import data and make sure that these are well tested. 

Q: “How do you recommend we use CD to synchronize, deploy and verify complex projects with database,  Agent Job,s SSIS packages and SSRS reports.” 

Dave: I would automated the integration of new packages as part of my deployment pipeline. I would also look to create automated tests that verify each change, and run these as part of my pipeline. 

Q: “How would you deal multiple versions of a database (e.g. development , internal final test, and a version for customer), and do you have any advice for the automatic build and deploy of a database?” 

Dave: I recommend the use of the ideas in “Refacoring Databases” by Scott Ambler and Pramod Sadalage. 

Q: “Do you have any tips for enabling rapid DB ‘resets’ during build/test? E.g. How to reset DB to known state before each test?” 

Dave: A lot depends on the nature of the tests. For some kinds of test, low-level unit-like tests it can be good to use the transactional scope for the duration of the test. At the start of the test, open a transaction, do what you need for the test, including any assertions, at the end of the test abort the transaction. 

For higher-level tests I like the use of functional isolation. Where you use the natural functional semantics of your application to isolate one test from another. If you are testing Amazon, every test starts by creating a user account and a book. If you are testing eBay every test starts by creating a user account and an auction…. 

You can see me describing this in more detail in this presentation – I am speaking more generally about testing strategies and not specifically about the DB, but I think that the approach is still valid. https://vimeo.com/channels/pipelineconf/123639468 

Q: “I’m concerned about big table rebuilds not being spotted until upgrade night.  Also obscure feature support like FILESTREAM. Do you have any tips for avoiding these kinds of last-minute surprises or dealing with a wide mix of systems?” 

Dave: I tend to try to treat all changes the same. I don’t like surprises either, so I try to find a way to evaluate every change before it is released into production. So I would try to find a way to automate a test that would highlight my concerns and I would run this test in an environment that was sufficiently close to my production environment to catch most failures that I would see there.  

Q: “Do you have any advice for achieving zero down time upgrades and non breaking on-line database changes?” 

Dave: I have seen two strategies work. They are not really exclusive of one another. 

1) The microservice approach, keep database scoped to single applications and create software that is tollerant of a service not being available for a while. I have done some work on an architectural style called “Reactive Systems” which promotes such an approach. 

2) Work in a way that every change to your database is additive. Never delete anything, only add new things, including schema changes and transactional data. So ban the use of UPDATE and DELETE 😉 

Q: “Managing db downtime and replication during CD” 

Dave: See comments to preceding question 

Q: “How do you craft data repair scripts that flow through various development environments?” 

Dave: I generally encode any changes to my database as a delta. Deployment starts from a baseline database image and from then on changes are added as deltas. Each copy of my database includes a table which records which delta version it is at. My automated deployment scripts will interrogate the DB to see which version it is at. It will look at the deltas to see which is the newest and it will apply all of the deltas between those two numbers. This approach is described in more detail in Pramod and Scott’s book. 

I think of the delta table as describing a “patch-level” for my DB. So two DBs at the same “patch-level” will be structurally identical, though they may contain different transactional data. 

Q: “What are some of the community-supported open source C.D. applications that would work well for an enterprise org that currently doesn’t have C.D.?” 

Dave: If you are going to take CD seriously you are going to want to create a pipeline and so coordinate different levels of testing for a given release candidate. So build management systems are a good starting point, Jenkins, TeamCity and Go from ThoughtWorks are effective tools in this area. 

I think that the tools for automated testing of DBs are still relatively immature, most places that I have seen use the testing frameworks from application programming languages and grow their own tools and techniques from there.  

RedGate have tools for versioning DBs. I haven’t used them myself, but they have a good reputation. My own experience is that up to now I have used conventional version control systems, Subversion or GIT, and stored scripts and code for my DB there. 

Q: “Which tools make CD (in general, and for the database) easier?” 

Dave: See above. 

Posted in Agile Development, Continuous Delivery, Effective Practices, External Post | Leave a comment

Pair Programming – The Most Extreme XP Practice?

Posted on April 4, 2016 by davef

I was an early adopter of Extreme Programming (XP). I read Kent’s book when it was first released in 1999 and, though skeptical of some of the ideas, others resonated very strongly with me.

I had been using something like Continuous Integration, though we didn’t call it that, in projects from the early-1990s. I was immediately sold when I saw JUnit for the first time. This was a much better approach to unit testing than I had come across before. I was a bit skeptical about “Test First” – I didn’t really get that straight away (my bad!).

The one idea that seemed crazy to me at first was Pair Programming. I was not alone!

I had mostly worked in good, collaborative, teams to that point. We would often spend a lot of time working together on problems, but each of us had their own bits of work to do.

With hindsight I would characterise the practices of the best teams that I was working with then as “Pair Programming – Light”.

Then I joined ThoughtWorks, and some of the teams that I worked with were pretty committed to pair programming. Initially I was open-minded, though a bit skeptical. I was very experienced at “Pairing – Light”, but assumed that the only real value was at the times when I, or my pair, was stuck.

So I worked at pairing and gave it an honest try. To my astonishment in a matter of days I became a believer.

Pair programming worked, and not just on the odd occasion when I was stuck, but all the time. It made the output of me and my pair better than either of us would have accomplished alone. Even when there was no point at which we were stuck!

Over the years I have recommended, and used, pair programming in every software development project that I have worked on since. Most organisations that I have seen try it, wouldn’t willingly go back, but not all.

So what is the problem? Well as usual the problems are mostly cultural.

If you have grown up writing code on your own it is uncomfortable sitting in close proximity to someone else and intellectually exposing yourself. Your first attempt at pair-programming takes a bit of courage.

If you are one of those archetypal developers, who is barely social and has the communication skills of an accountant on Mogadon1 this can, no-doubt, feel quite stressful. Fortunately, my experience of software developers is that, despite the best efforts of pop-culture, there are fewer of these strange caricatures than we might expect.

If you are a not really very good at software development, but talk a good game, pair-programming doesn’t leave you anywhere to hide.

If you are a music lover and prioritise listening to your music collection above creating better software, pairing is not for you.

There is one real concern that people ignorant of pair-programming have,  and that is “Isn’t it wasteful?”.

It seems self-evident that two people working independently, in-parallel, will do twice as much stuff as two people working together on one thing. That is obvious, right?

Well interestingly, while it may seem obvious, it is also wrong.

This may be right for simple repetitive tasks, but software development isn’t a “simple repetitive task”. Software development is an intensively creative process. It is almost uniquely so. We are limited by very little in our ability to create the little virtual universes that our programs inhabit. So working in ways that maximise our creativity is an inherent part of high-quality software development. Most people that I know are much more creative when bouncing ideas off other people (pairing).

There have been several studies that show that pair-programming is more effective than you may expect.

In the ‘Further Reading’ section at the bottom of this post, I have added a couple of links to some controlled experiments. In both cases, and in several others that I have read, the conclusions are roughly the same. Two programmers working together go nearly twice as fast, but not quite twice as fast, as two programmers working independently.

“Ah Ha!”, I hear you say, “Nearly twice as fast isn’t good enough” and you would be right if it wasn’t for the fact that in nearly half the time it takes one programmer to complete a task, two programmers will complete that task and do so with significantly higher quality and with significantly fewer bugs.

If output was the only interesting measure, I think that the case for pair-programming is already made, but there is more…

Output isn’t the only interesting measure! I have worked on, and led, teams that adopted pair-programming and teams that didn’t. The teams that adopted pairing were remarkably more effective, as teams, than those that didn’t.

I know of no better way to improve the quality of a team. To grow and spread an effective development culture. I know of no better way, when combined with high-levels of automated testing, to improve the quality of the software that we create.

I know of no better way to introduce a new member of the team and get them up to speed, or coach a junior developer and help them gain in expertise or introduce a developer to a new idea or a new area of the codebase.

Finally, the most fun that I have ever had as a software developer has been when working as part of a pair. Successes are better when shared. Failures are less painful when you have someone to commiserate with.

Most important of all the shared joy of discovery when you have that moment of insight that makes complexity fall away from the problem before you is hard to beat.

If you have never tried pair programming, try it. Give it a couple of weeks, before assuming that you know enough to say it doesn’t work for you. If your manager asks why you are wasting time, make an excuse. Tell them that you are stuck, or just “needed a bit of help with that new configuration”. In the long run they will thank you, and if not find someone more sympathetic to what software development is really about.

Pair programming works, and adds significant value to the organisations that practice it. Give it a try.

1A Medical Drug used as a heavy sedative.

Further Reading:

‘The Case for Collaborative Programming’ Nosek 1998
‘Strengthening The Case for Pair Programming’ Williams, Kessler, Cunningham & Jeffries 2000
‘Pair Programming is about Business Continuity‘ Dave Hounslow

Posted in Agile Development, Culture, Effective Practices, Pair Programming | 2 Comments