Do you find that your projects always take much longer than you thought they would when you started out? Well, don’t worry: you’re not alone.
Daniel Kahneman is a Nobel Prize-winning psychologist. Early in his career, he was working with a team of academics to produce a textbook when he had an idea for an exercise to do during the planning stages. He asked everyone to write down, independently, how long they thought it would take them to complete a finished draft of the textbook. He found the estimates were roughly centered around two years: the shortest being a year and a half, the longest two and a half years. He then had another idea. He turned to the curriculum expert on the team and asked him whether he could think of teams similar to theirs who had developed a similar textbook from scratch, and how long it had taken them to finish. When he thought about it, the curriculum expert said that a substantial fraction of similar teams had never finished the job - around 40%, in fact. And of those who had completed the project, most of them took more like seven years.
This information should have made the team drastically increase their estimates of how long the project would take, and maybe even rethink doing the project at all. But did it? Nope - they were optimistic about their chances compared to the other teams, and hey, they had a detailed plan which suggested they could finish in just two years. And so they carried on.
Guess how long it took them to complete the project?
Eight years.
It turns out that even Nobel Laureates-to-be easily fall prey to what’s known as the planning fallacy: a tendency to systematically underestimate how long things will take to complete. The problem is that we naturally think of best-case scenarios and what will happen if everything goes to plan. It’s hard to imagine all the different ways in which your project could go wrong or you might get held back, partly because these things are hard to see in advance, and partly because it’s just not very nice to think about the worst-case scenario. Interestingly, one study found that if you ask one group of people for their realistic “best guess” scenario prediction, and another for their optimistic “best case” scenario prediction, the two groups produce indistinguishable results.
Thankfully, there’s quite a simple way to guard against the planning fallacy. Think back to Kahneman and his colleagues. What should they have done? They should have taken into consideration what the curriculum expert told them about other similar teams, and adjusted their estimate accordingly, to something closer to 7 years. This is called taking the outside view: thinking about what happened in other, similar cases in the past, and basing your judgment on that. The mistake (though it’s tempting) is to take the inside view: to think your case is different from all those other cases, and just focus on the specific reasons you should be more optimistic. So if you’re trying to predict how long it’s going to take you to write that essay, you’re much better off thinking about how long similar essays have taken you in the past than focusing on how well prepared you are for this specific essay.
Of course, even if Kahneman and his team had taken the outside view, they still probably would have underestimated. The outside view said their textbook would take around seven years, when in fact it took eight. This might be because of something called Hofstadter’s law, which states that things always take longer than you expect, even when you take into account Hofstadter’s law...
