Managing expectations on time estimates with probabilistic forecasting

Project stakeholders always want to know when it will be done. So, we provide a time estimate somehow. And when it takes longer, we start wondering how we can get time estimates right in the future. But, can we?

I believe that’s not the question we should be answering. Estimating is uncertain and the most likely outcome is that we will get it wrong. So why are we even trying?

In my opinion, the problem to solve is not how to improve our time estimations, but how to make people understand that whatever date we name is not a fixed deadline.

We need to find a way to manage the expectations on the time estimates we provide.

Adopting a probabilistic forecasting approach can be of great help here. It relies on the following two basic principles:

1. The forecast consists of a time range and a probability that provide the level of confidence in the provided distribution. There is no single date based on an average or on a gut feeling.
2. The forecast needs to be updated regularly as more information becomes available. It’s a living forecast. It can change.

For the past five years, we have been using probabilistic forecasting in one of the departments of the global IT organization at Kuehne + Nagel in Hamburg. I have had the opportunity to work in one of the teams that follow this approach for providing time estimates. In this article, I will describe how the process works, how the expectations of our stakeholders have been influenced by it, and why this has been beneficial to everyone involved.

Creating the forecast

To generate the forecast, we use a Monte Carlo simulation. This is a mathematical tool that uses historical data as input to generate the probable outcome of future projects under similar circumstances. This video provides a good explanation of how the simulation works. The way we have used it in our forecasting process consists of the following steps:

1. Define the number of work items. The usual ‘task break-down’ session where the whole team sits together and comes up with an initial estimation of the steps needed to implement the requirement/feature.
2. Get the team’s historical data. This is provided as an input to the Monte Carlo simulation. We usually take the past six months of data, but this can be adjusted as necessary.
3. Run the simulation and evaluate the results. The result will be a curve looking more or less like this:

In this example, the curve is telling us that the team is going to finish 8 work items with 85% probability by February 25th or earlier. Note how the estimated latest delivery date varies, together with the level of confidence chosen to be communicated, as part of the forecast.

FAQ 1: What if there is no history available?

Forecasting probabilistically requires some record of the team’s performance. But of course, if the whole setup is new (people, project, maybe even the company itself) and there is no history yet, it will be necessary to use another method to provide an initial forecast (like Planning Poker, etc.). The idea would be to switch to a Monte Carlo simulation once there is some data available. Luckily, the vast majority of established software development teams in any context/industry nowadays do have vast amounts of performance data available. It’s time to make use of it!

FAQ 2: Do all work items need to have the same size?

No, they don’t. But what we have observed is that when a team has certain ‘flow stability’, which is also a result of cutting the work items as small as possible (or at least ‘similarly’ small), then the forecasts become more accurate. In any case, this doesn’t really matter if the forecast is repeated regularly. This brings me to my next point.

Getting along with the probabilities

It’s a fact that we humans don’t deal well with probabilities. Accepting these kinds of forecasts is rather hard, as my experience shows. More often than not, higher management takes the date and ignores the probability. Getting along with it requires a shift in the mindset: we need to think ‘probabilistically’ and not ‘deterministically’. This means we need to acknowledge that forecasting is an action in which more than one result is possible. Because forecasting is uncertain, we can’t see the future after all.

So, how do we deal with that uncertainty? Well, we take a probabilistic forecast as the initial answer, but we repeat it as we know more – as we get more information.

Continuous forecasting

Repetition is the key. This is the optimal way to get used to the probabilistic approach because it is possible to see how the forecast changes, which explains the meaning of the provided probability. The influence of external factors on the estimated date over time can be clearly seen as we re-forecast and the date changes. We get used to the fact that it’s a living forecast.

For us, it’s been a long way toward educating our stakeholders on the fact that the estimated delivery dates can change. It hasn’t been easy, and it’s not yet completely accepted by all of them. But having established a process where we regularly sit together and revise the forecast has opened many doors. The tradeoffs on scope and time are now data-driven decisions made with an explicit level of confidence: the probability percentile.

Enjoying the benefits

We haven’t spent time on detailed, item-based time estimations in the team for many years now. Instead, we just count the work items and throw them in the Monte Carlo simulation to get an estimated date. The provided forecast is much more solid because it uses our actual performance data as a base and we are then able to quantify the risk. This allows planning with a conscious level of confidence, and favors the transparency of the decision-making processes together with the stakeholders.

Moreover, continuous forecasting has enabled us to move away from just ‘working the (fixed) plan’. The fact of the matter is that ground conditions constantly change and we should react to that. At some point, the tactics have to change based on the new information we get. By following a continuous forecasting approach, we are able to do that in a natural way.

Conclusion

Estimations can be wrong; they are not fixed values. People need to move away from expecting an estimate to always be true or 100% accurate. By adopting a probabilistic forecasting approach, it becomes possible to start managing the expectations on time estimates. The course of development becomes clear, and it’s possible to understand how and why deviations happen. This, in turn, allows you to react on time, and the time/scope tradeoff decisions can be made in a data-driven way.

I truly recommend this approach. In my experience, it helps remove unnecessary pressure from teams, keep the focus on what really matters, and ultimately, manage projects in an agile way.