This article was written in 2005 and published in ‘International Developer’ magazine.
If there has been one constant in our industry it is the complaint that we are unable to reliably deliver software that is of a suitable quality, at reasonable cost and that meets the needs of our users. This complaint has been voiced since the early 1970’s, and it remains a defining issue of our industry.
Studies of productivity vary, but most seem to agree that for all but the smallest projects more than 70% fail to deliver what they promise on time. Why such a dismal success rate?
Agile developers believe that the processes that underlie most software projects are badly misaligned with the reality of software development.
Ultimately most projects are based on an illusion of science; while they aim to be rigorous, in reality they are largely grounded on qualitative measures.
Let’s imagine what it would be like to use the system and define our requirements on that basis.
Let’s guess how many people may use this software and how frequently.
Let’s imagine a design that will be able to deliver these requirements.
Let’s guess how long it will take us to build all of this.
All of these activities are speculative; they are attempts to look ahead and predict the future based on the guesses of experienced people. The reason that so many projects attempt to predict the future in this way is fear of the cost of change (See Figure 1). It is commonly understood that the later a change is made in the life of a project the more it costs. Traditional projects mitigate this effect by attempting to eliminate risk by foreseeing problems and identifying solutions to them as early as possible.
Figure 1 – Traditional development cost of change model
Agile development practitioners believe that planning too far ahead is doomed to failure, because, however good we are as software developers, we are not prescient and we will make mistakes.
Agile processes accept that mistakes will happen, that we will, change our minds, misunderstand something and generally behave like human beings. Instead of mitigating risk by trying to anticipate problems, agile projects are designed to avoid the increase in the cost of change completely. By accepting that change will happen and relying on several techniques to reduce the impact of late changes, agile processes alter the cost of change curve (Figure 2).
Figure 2 – Agile cost of change model
There are several techniques used on agile projects to effect this reduction in the cost of change. Instead of taking a qualitative approach to development, agile projects try as much as possible to take a quantitative approach. Our estimates for this week are based on what we did last week. Our progress is measured by how much software is actually working, not on how far we think we are through the problem. The utility of our software is measured by users using the software to do real work and telling us if it meets their needs.
These are pragmatic, measurable things. Of course, there remain times when we must make our guesses based on experience. Agile projects attempt to reduce the impact of getting those guesses wrong by not extrapolating too far ahead, typically only one or two weeks ahead.
Agile proponents try to avoid being too proscriptive about the nature of the processes used – people over process is one of our tenets. We aim to tailor the details of process to the needs of each project and team individually. The process described here is common to many of the projects I have worked on and is in part a form of Extreme Programming as described by Kent Beck. Typically these projects have had other attributes culled from other agile approaches, but they are beyond the scope of this introductory article.
The remainder of this article describes some of the techniques that allow us to modify the cost of change curve, and establish more quantitative development practices.
Requirements
Most people can imagine the software system that they want, but it is very difficult to explain such a vision in an unambiguous way. We could try to be more specific about requirements, or more rigorous in our description of how things should be, but a more agile approach is to ensure that the people who understand a problem talk to the people who know how to find a solution.
We write stories about uses of the system to capture small interactions that describe what the user wants. These stories are written in plain English, describing the problem to be solved. The stories don’t attempt to describe how the problem should be solved, and they don’t attempt to describe every detail of the problem.
Requirements, expressed as stories, aren’t enough to write code from, they don’t contain enough detail. Instead they represent place-holders for a conversation between a developer and someone who will use the system, or someone who represents the users of the system.
Stories are key aspects of our approach to agile projects. They represent the smallest piece of business value that makes any sense to the users of the software, and they are the currency of the project.
Stories identify requirements, define units of development, are the sole measure used to track the progress of the project and represent the scope of provable deliverable behaviour of the system.
We often use stories in the form shown here, but the form isn’t proscribed:
e.g.
Retrieve Contact Details
As a salesman I would like to be able to retrieve contact details so that I can make a sales-call with the details to hand.
I will know that I have achieved that
when:
I can retrieve the details for a contact where I specify their name,
I can see the details of the contact for the contact(s) I have found.
Requirements in this form are easy to understand by everyone. Any developer can build something that does this without the need for a more detailed specification if they are allowed to have a conversation with a user to clarify what is really needed and they can use the success criteria described in the story as a specification for tests. These tests can then be used to prove that the story meets the requirements.
Once a story has been developed the behaviour is demonstrated to users. At this point the users are free to change their minds. Perhaps the business circumstances have changed. Perhaps now that the users can see the real software, they perceive the problem differently and they have some new ideas. Perhaps we misunderstood the requirements and developed the wrong thing. In agile projects we accept that all of these changes may happen, and we structure our projects to not only support them, but to expect them, at any stage. Our processes allow people to be humanly fallible!
Documentation
People are the key to software development. Regardless of tools, technologies or techniques, software development is a human activity. Choosing the right people, ensuring that teams work effectively together and encouraging human interactions is what ensures that good software is the result.
Agile projects encourage people to work together collaboratively and to share responsibility for finding solutions to problems. They avoid processes that encourage the use of documents, diagrams or models that get in the way of human interactions. This is not to suggest that agile developments are document free, they are not. However, the aim is to produce the documents that are useful, and to avoid spending lots of effort developing documents that are either rarely or never used.
One key piece of documentation of the behaviour of the system is the collection of automated tests. Software developed in an agile project often has a much higher degree of automated testing than software developed by more traditional processes. This is a result of the pragmatic, quantitative approach that underlies agile processes. Software that passes the tests is finished; software that has no tests is not because we have no measure to show that it is finished.
This body of tests document both the behaviour of the system and its design. We generally find that in helping us to identify problems later in the life of a project, the tests are much more valuable than most design documents. The tests are guaranteed to be in step with the code, and we can therefore use them to confirm that any changes that we make preserve the existing behaviour and design of the system.
Tracking Progress
Stories are used as the atomic unit of progress in agile projects. In more traditional projects, progress is often measured by how many tasks are complete, or progress against a particular task;
“I am 80% finished on this task, and this task represents 5% of the total project so we are 4% of the way through the entire project.”
These are very difficult things to measure accurately. Any developer’s estimate of effort remaining is by definition subjective, and the break-down of tasks is based on estimates and design assumptions that are, again, only qualitative. This is a house of cards on which to build planning assumptions.
In agile projects we measure software in terms of stories delivered, functionally complete, fully tested, and deployed into production.
We estimate for small pieces of work over short durations; we do it frequently, every iteration, and we collect statistics on our accuracy. This means that we become practiced, and so consistent, in our estimates. Consistency is more important than accuracy. If we are consistent, even consistently bad, we can use the measures of our accuracy to allow us to understand, and predict over short periods, the rate at which we can build working software.
This approach to tracking has several important implications:
- We can’t sustain lengthy up-front analysis and design phases, because the delay between this and working software, our only measure of real progress, is too long, allowing the cost of change to increase. Instead we must find a way to evolve our designs as we go.
- The systems that we build must be working all the time so that we can avoid nasty surprises. This means that we must be able to prove that any changes we make don’t break existing behaviour and we must be able to reliably produce a working build all the time.
- In consequence, we take automated testing very seriously. Automated testing on agile projects is widespread and pervasive. It provides the principal defence against increases in the cost of change. Our ability to run regression tests quickly and effectively gives us confidence that the code we add or change does not compromise the fitness for release of the software.
- We must build the system often, and we will probably be deploying the system often too, which implies that we need to become very good at build and deployment, usually automating as much as possible to achieve a reproducible, reliable process.
We use processes like Iterative development, Continuous Integration and Test Driven Development to help us achieve these goals.
Iterative Development
We divide our projects into tiny sub-projects called iterations. Each iteration is focussed on the delivery of a small number of stories and has all of the stages of a full blown project: analysis, design, development, testing, and in many cases even deployment. Typically such iterations are only one or two weeks long, limiting our exposure to the risks associated with predicting our future performance to only one iteration’s effort. (See Figures 3 & 4).
Our users always choose the stories we work on. They choose the most important, most valuable stories first. At the end of each iteration, the stories will be complete, tested, reviewed and ready to deploy into production.
Everyone on the team is very focussed on the delivery of these stories. We try hard to avoid planning ahead for things that may not happen, and we try to maintain the focus on the things that are most important to the users of the systems that we build.
At the halfway stage in a traditional development processes, even when everything is perfectly on track, half the money is spent, but half the value does not yet exist.
Figure 3 – Traditional Development Process
At the halfway stage in an agile development process, even when everything is going badly, the most important deliverables are already working and either in production or are ready to be deployed into production.
Figure 4 – Agile Development Process
Continuous Integration
Every time a developer commits a change to the version control system we re-build the entire system, run all of the unit tests, and often run a batch of functional tests too. If any of these tests fail, the developer who broke the tests drops everything until the build is working again. Such miscreants are not allowed to go to lunch, go home or work on anything else until the build is working for everyone.
Developers take keeping all of the code working very seriously. They work hard to ensure that all tests keep running, because their colleagues demand it of them. This discipline, in turn, ensures that the software runs all the time, is tested all the time, and is ready for use all the time.
Test Driven Development
To be able to deploy our software at the end of every iteration, we need to be certain we haven’t broken anything that was developed earlier. We also want to avoid wasting time on documentation that becomes redundant when the code diverges from the ideal world of design. We need to design and build our code within a single iteration, and not speculate on problems that may occur in future. Finally, we need to help developers who come after us understand what we were trying to do.
Test Driven Development (TDD) fulfills all of these goals. Instead of writing lengthy design specifications, developers write unit tests that prove that the code meets the success criteria. These tests capture the intent of the component by defining not only an interface to it, but also how that interface is expected to behave.
TDD encourages good requirements by focusing on what is needed not how to achieve it. It fosters good software design because it keeps things as simple as possible while meeting the requirements. It documents the behaviour of the system in a manner that can never get out of step with the code and is of more practical use to developers who come later. Finally, and more as a by-product, it builds a pervasive unit test suite which often provides levels of code-coverage almost unheard-of in non-agile projects coverage of more that 95% of all possible code-paths is not uncommon on agile projects.
Summary
Agile software development processes are an attempt to focus on the core problems in software development, accept them as realities and deal with them. They aim to encourage pragmatic, disciplined approaches to avoiding the problems inherent in assuming that the future is predictable. Agile development aims to play to the strengths of human creativity and the effectiveness of collaborative approaches that rely on open honest evaluations of problems that arise unexpectedly throughout the life of any project.
By adopting the collection of mutually supportive activities outlined here, agile projects provide more concrete measures that allow projects to be steered toward success, rather than aimed at success from a great distance. Agile projects are reactive, adaptive, evolutionary, collaborative and transparent in nature, and they are a more effective, human-scale approach to solving complex problems.