Monte Carlo analysis for product development

23 Mar 2020

I’d like to introduce you to a technique I’ve found really useful for product development, as well as real-life stuff.

It’s called Monte Carlo analysis, and it helps you explore and deeply understand formulas and metrics by combiniting distributions, rather than simple scalar numbers.

To set up the technique, let’s look at a really simple question from my Twitter days: how many Tweets are on screen at any time in TweetDeck?

I wanted to know this to do some performance work, but there could be reasons related to showing ads or to answer a design question about information density.

Let’s make some assumptions to start with:

Based on this, we can say:

Job done? Maybe not.

The trouble is, there are some big assumptions here: screen sizes vary hugely, of course, but also Tweets can have hugely varied heights because of different types of media.

Additionally, column-width is configurable from a slim 210px up to a positively width-y 350px.

So we’ve got a big margin of error here.

If, for example we choose a bigger screen of 2560x1440px, a slimmer TweetDeck column setup and some text-only columns with shorter Tweets:

How do we account for this? (By the way: let’s assume we can’t actually just measure this stuff, for the sake of example. Even though we could.)

Monte Carlo analysis is a technique that uses:

repeated random sampling to obtain numerical results. The underlying concept is to use randomness to solve problems that might be deterministic in principle.

The idea is simple:

So, for our example above, we can define some distributions:

We can then randomly sample from each to produce an output. But this is the tricky bit: randomly sampling by hand is pretty laborious.

To help, I’d like to introduce you to a tool called Guesstimate. It describes itself as a “spreadsheet for things that aren’t certain,” and I love it.

It does the sampling for you and has a really intuitive way of managing distributions.

Here’s a screenshot from an example that models our Tweets-on-screen question:

TweetDeck Monte Carlo

We have all the inputs modelled on the left, intermediary results in the middle, and the result on the right. It works like a spreadsheet: each cell contains a distribution, or a formula that combines input distributions (or single numbers). The arrows are showing how data flows through the model.

As you can see, it’s saying that we’d expect around 50 Tweets on screen, on average, but that also that there’s a right-skewed distribution of counts between 15 and 130:

Tweets on screen

This has given us a much more nuanced view of the situation: we need to account for a small number of Tweets on screen in most cases, but there will some small number of cases that can reach over 200.

This is useful for corner-case testing: we can see easily see the extreme or pathological cases we need to worry about.

There’s a range of public models to peruse on Guesstimate, showing some of the power available.

For example, Adrian Crockfort modelled the expected response times for a storage service with a caching layer:

Storage Service Response Time

He’s even used one of Guesstimates other functions — randomInt — to model the chance of a cache hit or miss, too:

Storage Service Response Time: Over-All response

That’s why you can see several peaks in the distribution: there are sampling choices being made for each entry in the output distribution.

In another example, someone has modelled the Drake equation: “a probabilistic argument used to estimate the number of active, communicative extraterrestrial civilizations in the Milky Way galaxy.”

Drake equation

You can go a bit nuts, too.

So what else, in the real world, might you apply this to?

I’ve looked at:

In particular, modelling the potential outcomes for my team at work gave us a really strong understanding of what we could expect to achieve in the quarter and, more importantly, what we should not expect.

It’s pretty cool.

From here you can: