Creative testing
budget calculator

Imagine we want to predict how often a pro basketball player is going to make their free throws.
If they take 10 shots and make all of them, their conversion rate is 100%. But is it reasonable for us to predict that they will continue to make 100% of their shots going forward? Probably not.
Luckily, there is a metric that helps us make predictions given what we already know.

Confidence is another way to describe probability. For example, a 95% confidence level means you are confident that 95 out of 100 times the estimate will fall between the upper and lower values specified by the confidence interval. The confidence interval is simply a range between two numbers that we feel we can accurately make a prediction within.

Based on what we know of our pro basketball player, we can predict that they will make anywhere between 1 and 9 of the next 10 shots. Not very helpful, right? Well, we need to collect more data. The more data we collect, the more confidence we gain and the smaller (and more accurate!) our confidence interval becomes.

Let’s say after collecting over 1,000 observations of our basketball player, we can now predict with high confidence that they will make anywhere between 7.4 and 8.1 of the next 10 shots.

The more we observe the better we become at predicting the future. And here, the same goes for your ad creative.

This calculator will work for either A/B or Multivariate testing structure.

If you’re currently only  A/B testing, enter the total number of ad variants you plan to test.

If you’re multivariate testing (bravo, BTW), just enter the number of variants in your largest variant group.

Ex. You are testing 3 headlines, 4 background images, and 2 layouts. There are 24 total variants from 3 separate groups. Since the background variant group is the largest, you would input 4 into MAX VARIANTS.