In Praise of the Waterfall

I have been following an interesting discussion on scrumdevelopment looking for case studies on Productivity improvements and ROI from Scrum. Roy Morien wrote:

“What I am always puzzled about is that in the history of software development, during which the Waterfall type approaches have been taken as THE way to develop systems, I have seen little, if any, real evidence of the effectiveness and efficiency of using them. I have also seen little demand for such evidence. The industry has adopted these approaches, and that’s that! What I have seen is a vast amount of evidence that these approaches do NOT ensure successful outcomes.”

Why was the waterfall adopted without appropriate rigor? Very simple, you don’t need a statistic or a study to understand something that you already know!

This may be a surprise to some people, but waterfall was a substantial improvement compared to its predecessor – which I’ll call ‘unstructured chaos’ for lack of a better term. Waterfall imposes constraints on the development process, so it can proceed effectively. Waterfall says:

1. Figure out the business requirements first
2. Don’t change your requirements while you are implementing
3. Test your code to ensure that it works

I have met a number of large companies for whom imposing this basic discipline was a tremendous improvement over unstructured chaos. So when a CTO is hesitant to give up his waterfall, it’s because s/he remembers (and probably paid for) the bad old days.

Waterfall fails because these constraints are impossible to uphold over the length of an entire project. 

Scrum and XP impose these constraints at the sprint level, and introduce an additional constraint: the time-box. Seen from this perspective, Agile is surprisingly similar to waterfall!

By processing small batches and producing “finished” functionality, the performance, reliability and predictability of the system improves dramatically (the Poppendiecks taught us that).

Iteration also introduces the opportunity for frequent reflection, which turbo-charges the improvement process. Agile groups can get much better very quickly and fruits of their labor are more closely aligned with customer needs. Regular reflection also has the side effect of making work much more fun.

So let us not condemn the waterfall. It has served us well. It was developed by real people in response to real problems, who did the best they could with the knowledge that was available to them at the time. We can do better now. So a moment of silence, and let’s get started with the next sprint.