There exists an interesting and not immediately obvious relationship between the volume (impressions, unique users) of a campaign and its performance. This generally occurs due to inaccurate measures of attribution. The concept is simple once you think about it. Basically, advertisers tend to blast ad’s all over the internet. The signal to noise ratio is extremely low for display ads (<1% CTR, <0.01% direct conversions), making optimization difficult in many cases. The ad space has devised various methods to increase the signal in their campaigns. A few of these concepts are “view-throughs”, attribution windows, “hover actions” and many others. As a data scientist you realize that while there is very definitely some type of branding effect in advertising, it is unfair to attribute every downstream purchase to “branding”. While there is nothing a-priori wrong with adding in these types of metrics, it is their interpretation that is tricky. One of the first things you need to correct for in these scenarios is what I dub “campaign gravity”. Larger campaigns receive more credit (and thus higher performance) merely due to their size and not their effectiveness.
Let’s run through a thought experiment and see what this looks like: Suppose I’m running my e-commerce site, awesomewidgets.com. For some reason I decide to run a very large ad campaign of just public service announcements (PSA’s) that have no reference to my awesome site. At the same time I launch another ad campaign which is 10 times smaller with similar PSA’s. I would expect the performance of these two campaigns to be identical. After all, I’m not advertising my site and there should be no reason these campaigns would drive any incremental lift to my sales. But, when I run the numbers I see that the larger campaign is “outperforming” the smaller one by about 10%. Let’s take a look at why.
From the Universe of all users (or cookies, however you want to think about it) on the internet there are a few people that converted on my awesomewidgets.com.
I launch both of my PSA ad campaigns and I have the following situation:
What I’m really interested in is the intersection of all 3 circles. Users in this space have seen ads from both campaigns and converted. In a last touch attribution model, the way these campaigns are “credited” is via a race condition to see which campaign they saw last. A user’s history in this case might look like: A,A,B,A,A,B,A,A. The question is how often is campaign A versus campaign B last in a user’s chain? The answer involves the ratio of volumes for each campaign.
Enough with the pretty pictures, let’s do the math:
- Cost of campaign A: $1000
- Cost of campaign B: $100
Suppose that in isolation they both have the same “CPA” of $1.00 (1000 actions on A and 100 on B). Note that this CPA is basically fictitious since I’m not driving any actions to my site. This is the latent baseline CPA of any campaign. However, when I look at them together they have different measured performance.
- For every converter that sees ads from both campaigns, A “steals” about 90% (actually, 1-1/11) of the attribution away from campaign B
- If 10% of converters have seen ads from both campaigns, A “steals” 90% of this 10% overlap which is 9%
What this means is that 9% of B‘s conversions are now being credited to A and B will appear to be performing worse. So, of the 100 actions that in isolation are attributed to B, 9 of them will now go to A and campaign B‘s CPA will go up. The CPA for campaign B will now look like $100.00/91 actions ~= $1.10. A 10% reduction in measured “performance”!
What does this mean? If you are comparing performance across multiple campaigns of very different sizes using a last touch attribution model then bigger always wins. This can make analysis tricky on the backend to say the least. These types of gotchas abound in all complex systems and care must always be taken when doing analytics.
There is another reason larger campaigns will appear better and it has to do with improper sampling by smaller campaigns. Look for a future post on that topic.