Everybody knows the I in INVEST: each User Story or Product Backlog Item must be independent of the others. This means that errors can cancel each other out. Too high on feature 1, too low on feature 2. Over the course of thirty or more features, things average out.
I have realized that another kind of independence is critical for accurate estimates. Just as important as the functional independence is the organizational independence of the team performing the work.
In a phased approach (“waterfall”), projects always share people. For example, last month, the architects were working on the design of project P1. Now they have handed that work off to the developers and start work on project P2. So if the architects have a delay in their part of P1, that will not just impact P1, it will also impact P2. If management re-prioritizes, taking people off of say P3 to support P1, then P3 is also delayed.
So a delay in one project impacts many other projects in the same organization. The company has a tightly coupled architecture. It is emotionally satisfying, because it gives the impression of high utilization at the expense of predictability and performance in executing projects. Estimation errors and delays cascade. Errors seldom if ever cancel each other out.
In Scrum, all the skills needed to bring the project from start to finish are present in each team. The team’s composition is stable with few unpredictable changes. So errors can cancel each other out and seldom cascade, because the dependencies between projects and teams are held to a minimum.
So if you want good estimates, decouple your projects from one another by establishing teams that work autonomously and together for the entire duration of the project. (In other words, you should be doing Scrum or something very similar to it 😉