The Opposition Argument
Let’s start off this post by exploring the arguments against emergent design. I often hear many different arguments and reasons to oppose emergent design, so I will share the few that I hear most often. First, I often hear that emergent design leads to wasted effort because building a product that only supports a small initial set of customer needs will later require the product development team to rebuild the entire product (or a substantial piece) in order to support more advanced customer needs. The wasted effort comes from the rework that the development team needs to perform in order to build the advanced features on top of the naively simple system. It follows that if the development team had asked about the advanced features before starting work, then the product could have been built from the beginning to support both the simple and more complex cases.
Almost every experienced developer has several horror stories that they can very quickly relate about a poorly planned project where they endured the hardship of throwing away hours of work because the project direction (though, seemingly not the customer’s needs) changed frequently and dramatically.
In addition to this, I also often hear that emergent design leads to poor system architecture because developers do not align their current efforts with a greater picture at the outset. Without steadfast direction, developers create code that works, but lacks cohesion and tends to feel more complex than necessary. After a long enough timeline of working without a big picture, the overall system architecture suffers.
Almost every experienced developer has several horror stories that they can very quickly relate about a poorly designed system that was impossible to update or maintain because the system lacked a cohesive overall system architecture.
While I have definitely heard many more arguments against emergent design, I have heard these two far more often than any of the others, so I want to explore these problems a little bit deeper.
Product Rework and Bad Design
I ask you to set aside any preconceptions for a brief moment while we try to discover the root causes of bad design and product rework. In my experience, product teams typically blame product rework and bad design on poor or inadequate initial planning. That is, if we spend more time on planning, or plan in a more skillful, careful, or comprehensive manner, then the outcome of bad design and rework can be avoided. This seems logical at first glance, so very few people give this a second thought, and attempt to solve these problems by bolstering planning activities.
Improved planning generally leads to better outcomes, which gives us the impression that we have solved the problems, until they reoccur.
But they do reoccur so frequently in software development that it has led some teams into a state we sometimes refer to as “analysis paralysis.” That means that the product development team cannot make any progress on the product because all of their time is spent in planning activities. The problem of bad design and rework cannot be completely solved by better planning, but both proponents and opponents of emergent design agree on one thing:
The best designs tend to emerge from product development teams that have a very deep and complete understanding of a problem and its solution.
So, if better planning does not completely prepare us for success, then what else must we do?
True Emergent Design
Great planning will only take a team so far because it is only one source of information and learning. In particular, planning typically revolves around certain constrained activities such as lengthy conversations with colleagues and customers, documenting decisions, drawing diagrams, and so forth. These important activities prepare us to begin working, but cannot guarantee a successful outcome.
If better planning could guarantee a successful outcome, then one could learn to ride a bicycle perfectly the first time without falling with enough up-front planning.
Now, before I incite a rowdy debate about the theoretical ability of a human being to ride a bicycle for the first time without falling, allow me to focus on a more relevant aspect of this problem: learning effectiveness. One way to learn and gather information about riding a bicycle is to sit on it and attempt to ride it. The experience we gain by attempting to ride helps us to quickly learn and become proficient at riding, though we may fall and hurt ourselves. Similarly, when building a software system, the experience of solving a small initial set of customer problems allows a product development team to learn more quickly and effectively how to create a better design.
Almost every experienced developer has several stories that they can very quickly relate about some software that they wrote and immediately after completing the work, had a much better idea how to solve the problem.
We have the best and most complete information after not only preparing to solve a problem, but actually solving the problem in an inferior way. Up-front design suffers from a lack of information derived from the experience of actually solving the problem. Therefore, after we have created a software system we have gained the experience necessary to envision a better design that solves our past and current problems more effectively. (Note: I did not say future problems!)
Applying Emergent Design
In order to make the process of applying emergent design effective at solving problems and building great, well-designed products, we must do something with our newfound knowledge. If we solve a problem in an inferior way and then proceed to leave the system in that state, then the only thing we have accomplished is successfully building a bad product. Teams that use emergent design as a technique for building a crappy product in half the time by cutting planning activities make me terribly sad and disappointed.
Conversely, I have had the privilege of working with many individuals and teams that used their knowledge and saved time to refactor the system architecture to reflect a far better design than they would have envisioned based on up-front planning alone. In my opinion, true emergent design requires great planning combined with real world experience and enough time and resources to refactor things to a better state as we go.