A Good Book!
I was looking through my old books recently and came across Wicked Problems, Righteous Solutions, a book on software engineering practices and paradigms. It’s a favorite — I still mention it when chatting about my vocation — but it’s been decades since I read it. On a whim, I decided to see how well it has held up.
(Note that although the book has two authors, Peter DeGtace and Leslie Hulet Stahl, it states that the views expressed are those of DeGrace. So in the following I attribute the book’s content to him.)
An old book
Make no mistake, this book is from a different time. It was published in 1990 and discusses the state of affairs in the 80’s and 70’s. That world was
- Pre HTTP, pre WWW, pre WiFi
- There was no “Agile”, although the “Scrum” development methodology is sescribed.
- There was no UML, no continuous integration, no continuous deployment, no dev ops.
- It was before the Great App Explosion: Laptops were novelties, there were no cell phones, and for the most part computer programs were run by specialists.
But over all the book does not come across as dated, other that frequent referrals to “structured programming”.
…read a long time ago.
I have absolutely forgotten some interesting tidbits. For example, the book opens with the historical anecdote about the bell ringers at medieval cathedrals having a high risk of death by electrocution, since they rang the steeple bells during storms, in an attempt to drive away the daemons causing the storm.
It also slipped my mind that the book closes with a description of the work of J. Jaynes, “The Origin of Consciousness in the Breakdown of the Bicameral Mind,” the entertaining notion that before humans developed writing, they did not experience consciousness as we do, but instead perceived their thoughts to be commands from their gods. I would have sworn that I learned of this theory reading Neal Stephenson’s Snow Crash, but apparently I was exposed to it earlier.
They had Scrum back then?
As much as I’m on board with short release cycles and iterative or spiral development methodologies, I’m not a big fan of Scrum as practiced. That’s too bad for me, since bastardized Scrum has become the universal standard for software development. My low opinion notwithstanding, it was fascinating to read about Scrum being used decades before the Agile Manifesto was published.
Mind you, the lingo has changed. Notably, the word “agile” does not occur in the description. Also missing are “user story”, “points”, “stand up”, and “sprint”. So in what sense is it Scrum?
The key to the historic Scrum methodology was small, empowered, self-organizing, interdisciplinary teams. They teams were “empowered” in that it was made clear that the assignment was of critical import, and the team was responsible. It was “self-organizing” in that the team was left to its own devices as much as possible. They were interdisciplinary in that they included systems analysts, coders, testers, doc writers, and sys admins. The key idea seemed to be that all members contributed to all tasks – requirements analysis, design, implementation, test, and documentation. The use of the word “scrum” was a reference to a rugby scrum, in which a highly motivate, cooperative group makes a sometimes wobbly path to a common goal.
DeGrace presents Scrum as a high-risk methodology with high potential gains, and most applicable for introducing new, disruptive products. That is the biggest difference between it and Scrum as currently practiced, which seems best suited for churning out uninspiring, so-so apps that satisfy a mundane checklist of requirements.
(See The Lean Mindset for a discussion on the tendency of Scrum to produce mediocre results.)
So just what is a “Wicked Problem”?
The book’s title is an attention grabber. DeGrace credits H. Hitttel and M. Webber with developing the concept of “wicked problem”, and quotes extensively from their 1973 paper, Dilemmas in a General Theory of Planning, when describing it.
The characteristics of a wicked problem are:
- A precise definition of the problem may not be available.
- There is not a deterministic algorithm for solving the problem.
- The problem space and solution space overlap, so that introducing the solution may change the aspects of the problem.
- Solutions are not correct or incorrect. Instead they are good or bad. More precisely, they are either good enough or they are not.
A real-world example of a wicked problem would be distributing a patch for a software product. After customers have attempted to apply it, the nature of the problem has changed. Ideally, the patch is effected and the customers are happy. If not, however, undoing the patch or applying a new one will address a somewhat different problem than the original.
DeGrace’s salient point is that the Waterfall Methodology is ineffective for solving wicked problems because it assumes that requirements can be stated with precision before a candidate solution is implemented and fielded. I think this is irrefutable. Furthermore, because of the the extreme difficulty of producing precise requirements, I think a spiral or incremental development methodology will usually outperform waterfall even if the problem is not wicked.
What else DeGrace got right
DeGrace is spot-on with many observations beyond the inadequacy of Waterfall for solving wicked problems. For example, given that software development is more craft than science, adopting new or holding to old approaches can be as much about socialization as measurement. With this comes a tendency to dogmatism about methodologies, and to claim improved results (timeliness, quality) without proof or measurement. DeGrace suggests that promises of benefits of various methodologies should be met with skepticism and where possible, empirically checked.
DeGrace also points out that early exposure of some functionality improve chances of project success. More accurately, he notes that a major problem of Waterfall and other large-scale methodologies of the 1980’s is that so much time elapses before there are any observable results. I imagine that this invisibility was a driver for so many intermediate, none-code deliverables mandated by procurement rules and development methodologies of that time. But as those who remember those times can attest, there is often a tenuous relationship between a collection of design artifacts and a working system. Yet producing and reviewing these artifacts has a heavy cost.
Code-and-go still rules
“Code-and-go” is what DeGrace calls pre-methodology, slap it together approach to coding. And it seems to me that today’s sprints and road maps are essentially just dressed up code-and-go. I’ve seen too many shops that scarcely have repeatable builds. All too often, management sets the feature list and deadlines before even consulting the development team. The challenge in software development is not replacing Waterfall with Agile, but instead whether there is any rhyme or reason at all in the way we develop software.
Is there a solution? Nothing cut and dried, but in my experience there are practices that reduce development time and improve product quality. They include:
- Engaging developers early in the requirements formulation.
- Educating developers in the problem domain.
- Incremental releases.
- Early demonstration of end-to-end functionality.
- Minimizing the volume of non-functional design artifacts.
- Code reviews.
- Automated tests.
I believe that support for all but the last two is at least implicit in Wicked Problems, Righteous Solutions. All I can say about the last two practices is that failure to follow them incurs a heavy price.