It's important to outline the structure of your upcoming multivariate creative experiment before building the actual creative. A good experimental structure will assure you find valuable results, including positive and/or negative outliers. A poor experimental structure can result in unclear data which leads to confusion and a feeling of wasted effort.
When planning a multivariate experiment, there are a few key structural components that you need to decide on. These components include your KPI metric, avg. CPA for that metric, budget, timeline, and number of variables. Use the components that you are sure of (such as budget, KPI, and CPA) to help decide on the other components. In general, the KPI, avg. CPA, and duration are usually known or quick to figure out, while variables and budget need to be discussed & decided on.
In our July 2020 Does Creative Matter experiment we followed the below structure.
With a total of 27 ads (3 logo x 3 font x 3 background), our budget per ad was $9.25 and our daily budget per ad was $1.32. This experiment resulted in 3,464 total clicks with an average of 123 clicks per ad and helped us discover what creative components are most important for our campaigns.
Getting a large sum of results is easy in click-optimized campaigns, but multivariate experimentation is also very important for conversion-optimization. The biggest difference comes in that the avg. CPA is generally much higher for conversions (purchases, leads, etc.) than for clicks. We've outlined some examples of different structures to help you understand multivariate experimentation for your brand.
It's best practice to spend at least 80% of your avg. CPA on each ad in your experiment for good data collection - meaning we should spend about $10 per ad here ($12 CPA * 80%). Using the known components and our spend per ad, we're able to outline that we need about 350 ads for this experiment ($3500 budget / $10 per ad). With that information, we can decide on the variable groups and number of variants for those groups - 5 texts, 35 product images, and 2 designs.
With an avg. CPA for purchases of about $28 and the creative variables already chosen, all we need to do is figure out our budget and duration. Following the 80% of CPA rule, we want to spend at least $22 per ad here. Using that, we can figure out our budget of about $1,200 ($22 per ad * 54 ads). Lastly, we simply chose a duration of 14 days.
For products with a very inexpensive CPA the best practice is to spend around $10 per ad. When it comes to things like App Installs or Clicks, a single result does not carry as much value as that of a single result in purchase campaigns. Therefore, it's important to try and maximize results for these types of campaigns.
In this example, the CPA, KPI, and duration were all decided on initially, but the Variables & Budget were not. We could either simply set a budget first, or begin choosing variables and then find our budget from there. We decided to test 3 texts and 5 images for a total of 15 ads, which brought our budget to $150 ($10 per ad x 15 ads).
Companies advertising products with a high CPA have two options for their experiments: spend more or run less creative variants per test. In this example, we have a preset budget of $2,000 and know our CPA, KPI, and duration. With a $70 CPA we should be spending at least $56 per ad ($70 CPA * 80%). From there, we know that we should run about 36 ads ($2,000 budget / $56 per ad) - so we outlined 3 text, 4 graphic, and 3 design variants.
High-ticket products shouldn't expect the best practices of multivariate experimentation to be any different for them, they'll just need to spend a large amount per ad to capture significant results. In this example, we wanted to test 18 ad variants (3 image * 6 text) and needed to spend about $720 per ad ($900 CPA * 80%). That experimental outline means we need a budget of $12,960 ($720 per ad * 18 ads).
For blogs, news, and similar types of media sources that are optimizing their advertisements for clicks or other low-cost results (reach, engagements, etc.), it's just as important to build a data-driven creative process as it is for any other brand. The main difference between these types of optimizations is that the CPA is often very inexpensive.
In this example, we have a CPA that is too low to use for our spend per ad, so we need to default to spending about $10 per ad. With a budget of $5,000, we can run 500 ads at $10 per ad ($5000 budget / 500 ads).
The rules and practices used throughout the examples above can be adjusted for every product with any goal. Just input the components that you have to decipher the components that you don't yet have. Here's a quick rules breakdown:
