Incrementality Testing. Why?
The last few articles I’ve written have been very specific and about iOS14, so I’ve decided to widen the lens and write more broadly about how anyone can run incrementality testing. This is a concept that I find to be immensely pivotal for any growth engine. In a sense, this is connected with iOS14 as more of these tests will be paramount to validate paid channels + campaigns with attribution degradation. This test — plain and simple — allows companies and growth teams to determine the effectiveness of their growth marketing initiatives.
It’s weird because when I started my first growth role, I vividly recall how much emphasis there was on incrementality from specific colleagues whereas it didn’t seem to matter to others. It was very much black or white — truly a stark distinction. However, this easily could have been a matter of prioritization because more and more started to pay attention to this (including myself) as we started to scale.
There’s so much to unpack with incrementality testing but I want to laser in on how to leverage the power of this test. No matter how big your growth team and the resources at your disposal, I’ll showcase how you can add this test to your growth arsenal.
so much to unpack with incrementality testing
Why incrementality testing
Whether you’re reading this because you’re just learning about incrementality testing or are an incrementality test veteran, the value of using this test is necessary to growth marketing. You can be spending thousands or millions on any given channel, but if your users would have converted regardless of that ad spend, was it worth spending that? This is why measuring incrementality for each channel provides companies with actionable insights on where to throttle or ramp spend.
Take the example of spending $2,500 on Facebook + $2,500 on Google for remarketing campaigns with the goal of subsequent purchases. If we ran an incrementality test and found that Facebook had 80 incremental conversions whereas Google had 110 — we would want to spend more of our budget on Google. So how do we figure out which channels are more incremental?
Easy test methods
It may seem daunting to set one of these tests up but I’ve broken it down into three easy buckets below:
How to set the test up + volume prerequisites
You’ll first want to determine which channel you’re running this on and what changes you’ll be making on there. Let’s use Facebook as our example for this test.
If you can afford to pause Facebook ads for~5 days, do that. This is the most surefire way to get cleanest data as to the effects of running Facebook in your growth arsenal
If you can’t afford to pause your Facebook ads for ~5 days and have enough volume (50+ conversions/week in multiple geos) you can use a geo exclusion. For example, let’s say we were running Facebook ads nationally and were driving 90 conversion in Los Angeles per week — we could exclude Los Angeles from all campaigns on Facebook.
During the test
As a general rule of thumb, you won’t want to change anything on your Facebook campaigns during the test period. This means bids, budgets, creative etc — basically anything that can skew performance and invalidate the results should be left untouched
On the rest of your channels you’re running (Google, Snapchat, etc) you’ll want to make sure you’re not making any changes on there either
Leveraging results
Let’s say we ran the test between Jan 15–21 (Friday-Thursday), we’ll want to use the previous week Friday-Thursday (Jan 8–14) as our pre-test numbers.
In the above example, we can see that during our test period between Jan 15–21, we still had 50 conversions with Facebook ads paused. Meanwhile, when running our Facebook ads we had 150 conversions. This tells us that Facebook is 200% incremental. It’s a simple example, but this analysis gives us a new baseline CAC that we can apply and compare with other channels.
Note: If possible, use internal BI numbers rather than relying on Facebook ads manager dash.
Wrapping up
If you want to truly measure your ad performance and distribute your budget in an increasingly methodical way, incrementality tests will be your best friend. My hope is that incrementality becomes synonymous with ubiquitous. It should be a tool in every growth marketing engine arsenal, whether the team is one or five hundred.
My hope is that incrementality becomes synonymous with ubiquitous.
I’ll leave everyone with one thought — if you’re running five channels with multiple touch points, do you truly know the impact that any specific one is driving without incrementality?
Want more?
The results from incrementality tests can be used to form channel scalars and multipliers — which can be used to make a case to cross-functional teams on the importance of marketing efforts, etc. If diving deeper and into the weeds on incrementality testing is something you would like to learn more about, please let me know and I’ll write a follow-up piece.
Jonathan is a former YouTuber, UC Berkeley alum and growth marketing nerd who's helped scale Postmates, Chime and various startups. Currently growth @Uber 🚀 www.jmstrategy.com