The Anatomy of an Experimental Organisation

I am a software developer. I see the world from that perspective. In reality though that is only one viewpoint. While it is important that we are effective at delivering software, what really matters is that we are effective at delivering business value.

When I describe Continuous Delivery to people I generally spend a fair amount of time impressing on them that it is not about tools and technicalities. It is not even about the relationship between developers and operations or product owners and testers.

Continuous Delivery is about minimising the gap between having an idea and getting that idea, in the form of working software, into the hands of users and seeing what they make of it.

This vital feedback loop is at the core of not just good development, but of good business too.

I have been lucky to work in, or with, several companies that I would describe as Agile companies, not just companies practicing Agile development. These organisations are fundamentally different in approach in almost everything that they do.

The principal characteristic of this kind of organisation is that they are experimental in their approach to everything.

For some more traditional organisations this sounds scary. “Experimenting, that sounds like you don’t know what you are doing”. Well, yes, we don’t know what we are doing, none of us really know!

To quote one of my heroes, Richard Feynman:

“It doesn’t matter how intelligent you are, if you guess and that guess cannot be backed up by experimental evidence – then it is still a guess!”

Research into the performance of successful organisations says that 2/3rd of their ideas that are implemented in software produce zero or negative value:

“Only one third of the ideas tested at Microsoft improved the metric(s) they were designed to improve”

“Avinash Kaushik wrote in his Experimentation and Testing primer that “80% of the time you/we are wrong about what a customer wants”

“Mike Moran wrote that Netflix considers 90% of what they try to be wrong”
(Ron Kohavi, Alex Deng, Brian Frasca, Toby Walker, Ya Xu, Nils Pohlmann 2013).


Two thirds, or more, of the ideas of GOOD companies are waste.

Further, the experience of these companies, and the data, says that nobody can tell which third of ideas are the good ones before they are delivered. Clearly there is a place for guessing and predicting customer demand for innovative ideas, but it is important to remember that for every iPhone that you invent, you will have to go through three or four Newtons first.

If we are generally so bad at guessing, then the only sane strategy is to embrace the uncertainty. Optimise to have lots of ideas and to evaluate them quickly so that you can discard the poor ones as fast, and cheaply, as possible.

This is what really effective organisations do. Watch Henrik Kniberg’s wonderful descriptions of the Spotify Culture.
See how Netflix work.

These companies know that their ability to guess is poor, because they are becoming more scientific in their approach. Instead of guessing what their customers want, and then assuming that their customers like what they are getting, they are measuring. These companies are designing experiments. They define metrics, that will identify what their customers think and then carry out the experiment and reflect on the results in order to learn and improve.

So how do these experimental companies differ from others?

Value Innovation over Prediction

Innovation is the thing that differentiates the great companies from the rest. Great companies create new markets, provide new products or services that change how people do things.

More traditional companies value predictability. The trouble is that you can’t be both predictable and innovative at the same time. They are different ends of a spectrum. The only really predictable thing is status-quo. So if you want your company to be great, you need to value innovation and discard your reliance on predictability – at least for some of your products.

One aspect of this is launch-dates. Apple don’t pre-announce their products, they are very secretive. Only when the products are ready do they announce. This allows them to strive for “Insanely great” as Steve Jobs so memorably put it.

State your hypothesis

This shared vision is kind of fractal. It operates at all levels in effective organisations. Being hypothesis driven helps to establish this clarity, this shared purpose. Useful hypotheses can range from, stating what you expect the outcome of a test to be before you execute it to stating how you think your global business strategy will work out.

Some people even talk about “Hypothesis Driven Development“.

Seek the right experiment

As soon as you have a new idea. Whatever its nature, the next question should be “How can we test this idea?”. If it is an idea about how to improve your process, figure out an experiment to see if it works: “Ok, let’s try doing without the estimation meeting for a while and see if we save time”. If it is an idea about improving your product figure out an experiment to test that too: “Let’s release this feature and A/B test it against the existing service to see which one makes more money”.

Establish Feedback Loops

Experimentation means nothing without the closure of the feedback loop. After each experiment there should be a way to evaluate the results and figure out what to do next. The outcome from each experiment should be a real action of some kind. Depending on the results of the experiment your should either: Drop the change; Adopt the change; Run through the cycle again with a new experiment to learn more.

The establishment of effective feedback loops is a vital attribute of experimental organisations. Continuous Integration, Continuous Delivery, Experiments in Production, Test Driven Development, Retrospective meetings and Incident Reviews are all mechanisms for establishing effective feedback.

Assume the fallibility of experts

All of the really high-performance organisations that I have seen are notable in their attempts to minimise hierachy. Decisions based on the highest paid person in the room are always only guesses! Seek evidence, allow individuals and teams the freedom to experiment and make decisions based on what they find. Treat people’s opinions with respect, but recognise that whoever it is that holds them, they are still only opinions. Test the hypothesis!

Question Everything

There should be no sacred cows. Everything should be up for evaluation, and if it is found wanting should be amenable to change. Technology, practice, process, office-space, team organisation, even personnel.

If it is not working, try something else.

Eliminate waste

Successful organisations avoid doing unnecessary work. The Lean mantra of eliminating waste is a powerful one. We should be rational and objective in assessing everything that we do in the light of our real goals and finding how to make our work more efficient.

The most common response that I get when coaching teams how to do better is “We don’t have time to improve”. Investing in making your team and their working practices more efficient is almost never a cost!

Leadership is Different to Management

Spotify talk of having “Loosely-coupled, tightly aligned teams” as being one of their goals. They aim for high autonomy and high alignment.

Dan Pink talks of the importance of “Autonomy, Mastery and Purpose” in motivating people towards excellence.

These ideas require leadership, not management. The goal of leadership should be to establish a common vision, a shared purpose for the organisation without telling people how to achieve it. Ideally leadership is about inspiring people.

My experience has been that experimental organisations are generally much more successful than their more traditional counterparts. They also tend to appeal to the most talented people, because of the creative freedom that they offer.

People sometimes ask me “how do you know when your organisation is working well?”, I think that you know things are on the right track when your first response to any challenge or question is “How could we try that out?” rather than “I think this is the answer”.

This entry was posted in Agile Development, Continuous Delivery, Culture and tagged . Bookmark the permalink.

2 Responses to The Anatomy of an Experimental Organisation

  1. amarmark says:

    Being experimental is pervasive, it introduces effective feedback cycles at all levels in your development process and for all kinds of different aspects of development, team and business organisation.

  2. Hey Dave, I agree with every single word. I am wondering tho, do you have any advice on how to convince the masses in companies, that this is not only the view of a nerdy developer, but a better approach in delivering products to customers and ultimatly increase revenue?

Leave a Reply

Your email address will not be published. Required fields are marked *