What is Modern Software Engineering?

I have a new book out. It’s called “Modern Software Engineering” and I have been working on it for the past few years. 

The ideas in it grew out of a growing realisation that the way that I approach software development, and the way that all of the teams that I was familiar with, that I considered excellent at software development, shared some fundamental characteristics. 

This got me interested in trying to nail down those characteristics and formulate them into a model that I could use to explain what it seemed to me worked best for software development.

Applying Science

I recognised that my own thinking has long been influenced by one of my hobbies, I like to read and learn about science. I am interested not just in the findings of science, but also in the organised approach to knowledge acquisition that it represents. Science is humanity’s best approach to problem solving, so it should certainly be applicable to a difficult, technical discipline like software development.

I had long described my preferred approach to software development, Continuous Delivery, as a simple, pragmatic, application of scientific style reasoning to solving problems in software. This led me to become really interested in exploring this idea in more depth. What does applying an informal approach to scientific learning and discovery mean when we apply it to solving practical problems? There’s a word for that, we call it “Engineering”.

Engineering != Bureaucracy

At this point I got rather nervous. It seems to me that in our discipline of software development the term “engineering” has become either incorrectly loaded with meaning, or emptied of it all together.

On one hand, many people assume that “Engineering” means stifling bureaucracy and heavy-weight process control. 

On the other, “Engineering” simply means writing code and nothing else.

Both of these are profoundly wrong. In other disciplines, “Engineering” is simply the stuff that works. It is practical, pragmatic and more, not less, efficient. 

Sure, it may offer some guide-rails, constraining our thinking, but it does that in a way that helps us to rule-out, or at least steer us away from, dumb ideas. This is a really good thing! 

Avoiding Bad Ideas

In software development we have a poor history of being able to eliminate bad ideas. We tend to make the same mistakes over and over again. 

Data scientists not using version control, resulting in 90% of ML projects never making it into production

Low-code environments that work on the assumption that you know exactly what you want at the start of a project and that that requirement will never change – good luck with that idea! 

So something that was able to steer us away from bad ideas would be a very valuable thing to have.

The Billion Dollar Mistake

The idea that “Engineering == Bureaucracy” is wrong, it comes from a completely incorrect, but understandable, mis-categorisation of what engineering in other disciplines is about, and then applying that mis-categorisation to software.

Humans are used to building physical things, so the production of physical things is front and centre in our minds when we think about making things. The inspiration and design of a physical thing is certainly a challenging problem, but it is so much more difficult to scale that up to produce those things en-mass, that we assume that that is where the only real challenge lies and so we assume that is all that engineering does.

We assume that “Engineering == Production Engineering” and so we, as an industry, made the billion dollar mistake of attempting to improve the efficiency of software development by applying production-line techniques. Didn’t work!

Production is not Our Problem

In software “production” is not our problem! Our product is a sequence of bytes, and we can recreate any sequence of bytes essentially for zero cost. 

This means that we NEVER have a production problem! Our problem is always one of learning, discovery and design. Engineering for software then, needs to focus very firmly on that part of the challenge and ignore, or at least automate our production process. 

Design Engineering NOT Production Engineering

So how do we optimise for exploration, learning and design?

If we want to look for examples outside of software, this is much more closely related to the innovative creation of new things, than it is to production engineering. Design engineering is a very different discipline. Think NASA designing Mars rovers, or Apple designing the first iPhone or SpaceX designing their Starship.

For this kind of engineering you optimise to be great at learning. Modern engineering consciously designs systems in ways that allow the engineers to iterate quickly and efficiently so that they can learn what works, and what doesn’t. We need to do the same.

Designing Complex Systems

The systems that modern engineers create are increasingly complex and sophisticated, so as well as focusing on learning, modern engineering in general, but certainly modern software engineering, needs to focus us on managing that complexity. We need to focus our tools, techniques and mindset on dealing with the complexity that is always at the root of our discipline. 

Software Development as an Engineering Discipline

I came to the view that this assumption that “software development isn’t really engineering” is correct in practice, but very wrong in principle

How I worked for most of my career certainly did not qualify as engineering, it certainly was closer to craft. However, in the latter part of my career I started to think more consciously about how I could do better. 

I started to take a consciously more rational approach to decision making in all aspects of software development. I started to apply some heuristics that would guide me, and the teams that I worked with, more reliably towards better outcomes. It works!

Modern Software Engineering

I have tried to outline this organised, but pragmatic and low-ceremony approach to software development in my new book. To capture some principles that I think are generic to all software development in a way that we can adopt them and use them to steer us in the direction of more successful outcomes. 

My thesis is this, if an engineering approach to software development doesn’t help us to create better software faster, then its wrong and doesn’t qualify as “Engineering”. 

Like most authors, I was nervous about how these ideas would be received, but “Modern Software Engineering”, my new book, is starting to gather some great reviews and people are finding the ideas in it as helpful as I have. If you read it, I hope that you enjoy it.

This entry was posted in Continuous Delivery, Continuous Integration, Culture, Effective Practices, Engineering Discipline, Personal News, Software Design, Software Engineering, TDD and tagged , , , , . Bookmark the permalink.

12 Responses to What is Modern Software Engineering?

  1. AC says:

    Hi,

    You say:

    Low-code environments that work on the assumption that you know exactly what you want at the start of a project and that that requirement will never change – good luck with that idea!

    Could you elaborate on this? Why do you think low-code is a bad idea?

    As far as I know, low-code doesn’t mean that the implementation / requirements cannot be changed.

    Thanks in advance.

    Best regards,
    AC

    • davef says:

      It is less that I think “Low code is a bad idea” and more that I think that the problem with coding isn’t what most people, at least most low-code creators, think it is. The problem with creating computer systems isn’t a problem of typing, it is that we have to consider all sorts of things that we are likely to get wrong first time. That means that we have to organise our work so that when we do get things wrong, we can find the problem, and fix it quickly. For that we need ideas like version control, and testing. Every low-code system that I have seen break down in nasty ways when you pass some threshold, that is hard to determine up front, of complexity. Big, broken, spreadsheets are a terrible, and regular source of errors in big firms. That is not because spreadsheets are evil, it is that they are an overly simplistic representation of the problems of programming a computer, and at that hard to see threshold of complexity, they aren’t maintainable. All low code systems suffer from this, as far as I can tell. They are great at simple, constrained, problems, and then they are a disaster when you pass that threshold.
      My view is that software development is considerably more complex than it looks on the surface, and is about lots more than the syntax of your programming language.

  2. Robert says:

    Dave,

    Just wanted to note that your book has brought back the joy of programming for me. I’m actually doing some programming for fun for the first time in a long time, because the approaches outlined in your book are so liberating and make even big projects seem approachable.

    Thanks,

    Robert

  3. Chad says:

    Hey Dave,

    Bought your book on audible and noticed it references an accompanying pdf that does not appear to be available through audible for some reason. Do you have any suggestions on where I may get it?

    Enjoying listening so far.

    • Chad says:

      Looks like they have added it to audible now!

    • davef says:

      I have let the publishers know, but I have yet to hear confirmation that this problem is now fixed. They say that each purchaser of an audiobook will get an email with a link to the PDF, but this isn’t yet in-place.
      I am very sorry, I think this is poor service, but it is not in my control. I can’t send you copies of diagrams myself, because the publishers own the copyright. Very sorry.

  4. derio says:

    Love your book, but the Audible version does NOT include a Companion PDF, even though it says it does in the description.

    This makes it impossible to follow through the examples. Please fix this!

    • davef says:

      I have let the publishers know, but I have yet to hear confirmation that this problem is now fixed. They say that each purchaser of an audiobook will get an email with a link to the PDF, but this isn’t yet in-place.
      I am very sorry, I think this is poor service, but it is not in my control. I can’t send you copies of diagrams myself, because the publishers own the copyright. Very sorry.

  5. DG says:

    Hi Dave,

    inspired by your writings and your well served youtube videos, I would like to share with you some thoughts on maintaining small/personal repos. I believe you might like it.
    https://github.com/cr3a7ure/gira

    Thank you for teaching us great principles,
    DG

  6. Stathis says:

    Hello Dave,

    I love your book so far and I could not wait to finish it before I message you.

    It came to my hands in perfect timing as it is the last 1-2 years that I understood that my focus needed to change if I wanted to be a better developer. Your book has already sifted my mindset on how I see programming in general. The concepts of the book are presented well and you manage to “unhide” them from the sea of information that is available for every developer! It is good to have an abundance of content but focusing only on shiny frameworks and languages is not what would make me any better. Your books show me the way.

    I was always struggling to get the “full image” of what an engineer should be before that book!

Leave a Reply

Your email address will not be published. Required fields are marked *