Help! I’m Trapped in a Waterfall!

Back in February, I talked about Project Management being about the “Art of the Possible.” Ken Schwaber invented this phrase and it really summarizes what project management, especially crisis project management is all about.

So what do you do, when the constellation of customer, schedules, partners and other third parties, and management makes a change of the fundamental working arrangements difficult or impossible? What do you do, when you are trapped in a waterfall?

I was confronted with this problem last fall when I took over a flagging project. It had serious organizational problems. It was a waterfall. My company was responsible for steps 2 and 3 of a 5 step process, but no one knew what really had to be done by when. People would start working on step n+1 before step n had been accepted by the customer. Even after formal acceptance, the customer would make changes to the graphics designs, so that nothing was ever really finished.

What to do? The problem was that there were four companies involved in the project: the customer, a usability specialist who did the functional spec, my company (graphic design and HTML publishing) and a software development company. My company, being the middle, had only limited influence on the overall organization of the project, so we were pretty much stuck with the waterfall, at least for the moment.

I decided to introduce Scrum (or what I like to call, “the Heart of Scrum”) while disrupting work and procedures as little as possible.

We started with a retrospective. This got everyone on board. There is no better way to earn a team’s loyalty than to start out by asking them what we should do and then doing it!

We identified a lot of potential improvements and got to work on implementing them. Turns out a high percentage of the issues were dealt with just by introducing Sprints, Self-Organization, Product and Sprint Backlogs, and a Definition of Done.

On big point was having a proper management of the deliverables: What do we have to produce for the customer.

At the begininng, I didn’t talk about Sprints, just about “Synchonization Points” (a term I encountered a few months later reading Tom & Mary Poppendieck’s excellent Lean Software Development).

We used Target Process to manage the requirements. TP is very good for agile Project Management but a bit challenged in the waterfall.

Were the requirements User Stories? Well TP called them that. But actually our basic unit was the Screen (one HTML Page). One published HTML Page. Each page was a user story. Pages were grouped into Features.

Since we were a step in the waterfall, we introduced a formal definition both for “ready” and “done”. “Ready” for graphic design was achieved when the customer had uploaded the wireframe into the corresponding graphic design story. Ready for publishing was achieved when the customer had uploaded the corresponding document into the HTML Story.

This was a real plus for TP: it’s easy to find and link information. (Another Lean Topic)

Our definition of ‘done’ included customer acceptance. That is, publishing wasn’t finished until the customer said, ‘yes, this is how I want it’

The advantages of this approach compared to the previous way of doing things were substantial. We stopped publishing designs that we still in progress. We got a clear understanding every sprint of what the customer wanted. We got a measure of velocity.

In preparing for the first sprint, we got our act together (‘our house in order’ as we say in German). By the third sprint we had substantial productivity improvements. Within three sprints, we had apparently dramatically improved our productivity and quality.

But there have been some problems. Most notably, the requirement that the customer approve each step has created a significant bottleneck: the customer. Furthermore, the approval process has not guaranteed that things that are declared ‘done’ are in fact done. They come back from the team downstream with issues.

A waterfall framework fundamentally means that requirements are pushed in to the pipeline and onto the following steps. Matching capacity across the teams is tricky and the risk of local sub-optimization is great.

[ Update: May 28: An even bigger problem with the waterfall is the cycle time. A Value Stream Analysis shows that the work to wait ratio for any given feature is probably between 1:6 and 1:10 – if the feature needs 20 working days (4 weeks) to realize, the calendar time could easily be 25 to 40 weeks, due to the wait times implicit in a waterfall model. ]

What did we accomplish? Was it a step in the right direction? Undoubtedly. The clarity afforded by scrum is priceless, even if you’re working in a waterfall. Have we done away with the problems of the waterfall? No. But the problems are becoming clearer, and we can continue to ask the “why?” questions and “how do we make this better?” questions. Sooner or later, the customer will be ready for change…