Something that analytical folks talk a lot about is “the incremental”. In customer marketing, it’s not enough to get response, what we want to know is how much of the response was above and beyond what we could be expected had the test or promotion not been done. Typically incrementality is measured using control groups – people just like the people you are sending the campaign to who do not receive the campaign. Then you compare the profitability of the people who received the campaign versus the profitability of those who didn’t. The difference is the true, incremental profitability of the campaign.
Kevin over at the MineThatData blog relates that Blue Nile has decided to lower prices as opposed to chasing rising ad costs. I don’t know enough about their business model to really comment on that action, but I do know one thing – they (and many, many other companies) are wasting Pay-Per-Click spend on non-incremental clicks.
Check out this shot. Search for Blue Nile in Google, you get a Pay-Per-Click ad for Blue Nile, and Blue Nile has the fiirst natural listing:
Â
Â
Â
Â
Â
Â
Now someone, somewhere, back in the beginning of PPC probably had a conference speech that told everybody to buy the search for your company’s brand name, maybe for “branding” or “exposure” purposes. But almost every web site ranks first in the natural rankings for their own brand name. So I ask you, is there any incremental here? What are the chances, if the PPC ad was not there, that the visitor searching “Blue Nile” would not click on the first natural listing?
What do you suppose the volume of this search is, and how much money is being wasted by people clicking on the paid ad who would have clicked on the natural link anyway? I asked the same question back in 2003, and though the structure of the test wasn’t as pure as most hard core test & control folks would like to see, the results were so dramatic that you can’t really argue much with the methodology. A few percentage points here or there and you still get to the same place.
In this particular test, on a high volume PPC phrase for a site that also had a top 3 natural ranking, 77% of PPC sales were non-incremental – stolen from the natural side – and would have happened anyway without the PPC link.
I will be the first to say that results are going to vary a lot by the type of site, the phrase, the target audience, the search engine, and so on. But Golly Gee folks, has anybody else ever questioned the wisdom of “railroading” a top natural listing with a paid listing – and especially for your own brand name? I bet a lot of analytical folks have, they just couldn’t convince the marketing folks to test it. After all, addressing / fixing this by taking down the PPC ad means:
1. There will probably be a few lost sales, perhaps from people who for whatever reason, always click on paid ads (newbies?)  However, profits will increase. Some folks will not believe in this idea, it’s that whole analytical culture problem. You have to trust the numbers, not fear them. Unless, of course, you have been specifically told to grow sales and not profits. Rare, but it does happen.
2. Then there is the whole “How could you not know this was a bad thing and continue doing it for so long?” problem.  That’s more analytical culture stuff. Failure is a learning experience in the analytical culture, not a cause for whippings and firings and demotions. The question is not how much money was lost, the question is how much will be saved going forward. Right?
3. Potentially, some PPC budget will be reallocated to other, more profitable marketing activities. This could be a problem if there is “ego spend” involved in the management of PPC. I wouldn’t move the money out of PPC, I would just find better ways to use the existing budget. But the potential exists for losing it for a higher and better purpose that is proven out by the numbers. Just optimizing the system, no hard feelings, right?
PPC Marketers, if you’d like to free up budget for something else, ask your web analytics folks if they can track this test for you.  Locate a high volume search phrase that you buy PPC for ads for where your web site naturally ranks as #1 or #2 for the same phrase.  Try shutting the PPC for that phrase off. How much click volume do you lose versus the cost of the clicks when PPC is tuned on, and what does that translate into in terms of increased profit?
And I am really not picking on Blue Nile here – search any brand name, online or off. Go ahead. What do you see?
Would you fail to click on the organic link if the paid link was not there? Really? If you think you’d miss the organic link if the PPC link wasn’t there, send me a pic of the example.
Welcome to the blogosphere, Jim. Glad you are maintaining a blog. Have fun!
Good points, Jim. Many firms that are marketing online fail to recognize that many consumers use search engines as navigational devices — NOT exploratory tools.
What about as a blocking strategy, to keep competitors out of the perceived “top spot”? I guess the numbers would still provide the guidance, but I can envision the category killer viewing these (generally minimal) dollars as money well spent to block an offer. I hear you can’t buy competitors brand names, but the enforcement of that seems very lax. Don’t get me wrong – I will be pursuing a test of this strategy and hope to save some money…
mb
Sure, I’ve heard the “blocking strategy” argument and like everything on the web, it probably makes sense in particular situations (Mortgages?). But if it still loses you money I wonder what this strategy is worth? What I’d really like to see is a financial proof that the “blocking strategy” pays out in the end.
Worse even than the blocking strategy is the “allocation strategy” where people simply “allocate” a certain amount of revenue to PPC and a certain amount to natural clicks based on some formula because “they work together”. This is by far the worst idea because it’s very easy to build a faulty case for PPC, especially if it is cannibalistic to natural. This is a very “old media” idea and symptomatic of the accountability problems in marketing these days.
I’m pretty much for any strategy, no matter what it is called, providing you can prove it is the most productive! Thanks for the comment Mark. Let me know how the test comes out, if you need any help with thinking it through let me know; this is the way I did it.
Hi Jim,
nice post.
you mention control groups in this article. Is there a way to apply the principle in PPC campaigns ?
Good luck with your blog !
K
Hi Kostis, and thanks for the post.
The answer to yur question is: perhaps but not really. This is a bit complex and I’m not sure how much you know about this topic, so I will lay it all out for the sake of being complete.
Control groups are most often used in customer marketing promotions to “control out†the response of customers who “would have bought anywayâ€. This gives you a better picture of the true effectiveness of a promotion. PPC campaigns are generally thought of as customer acquisition tools, and so creating a “control†in the traditional sense is really not possible since you don’t have a pool of “non-customers†that you can measure against to see “who would have become a customer anywayâ€. Follow?
So really what you have with a PPC test of this nature is more like what would be called A / B / C testing; you’re not testing versus a real control, but testing against the “lack of an adâ€, where A and B are PPC ads and C would be “the lack of A or Bâ€. Then you would compare results.
Now the problem with this approach is (as far as I am aware) you cannot scientifically exclude an ad in a true random fashion and keep track of that exclusion set, so you really can’t measure the results of the ad versus “no adâ€. Perhaps you could with some significant bid management / search engine API technology, but I’m not sure the engines would give up that kind of control – though I bet THEY have tested this at some level. It certainly would not be in their best interest to have people out there proving “no ad” was more profitable than “show ad”!
So here is what I did to approximate a true random sample exclusion test.
If you run campaign A on Monday, B on Tuesday, and C (no ad) on Wednesday, then start over with A on Thursday and continue this rotation, the following Monday B will run and on Tuesday C will run etc., so by the end of 3 weeks you will have A, B, and C data normalized by day – both campaigns and no campaign ran on every day of the week. Not a statistically pure method, but not horrible practice, for sure.
If the result spread (ad versus no ad) is significant enough, as it was in my PPC with Top 3 Organic link test, I’ll give up points in accuracy to get closer to the “directional truth”. Each of the top 30 search phrases where there was a top 3 organic ranking for the phrase was optimized in this way with the results very directionally consistent across all phrases. It was almost always more profitable to have a lower than #1 paid ranking when a top 3 organic ranking was present. Below the Top 30 phrases, some of the lower volume phrases produced inconsistent results which was probably a result of test method error / lack of frequency.
It probably would not be *technically* difficult to do a real random sample A / B / C test, if for example, Google allowed “No Ad” as an option under “Ad Variations†AND kept track of the visitors who searched with the PPC phrase who were *not shown* the ad. You would compare how many of these “no ad†visitors actually made it to your site / took the desired action versus those that were shown the ad, and then you would have a truly scientific test. But like I said, is it in the engine’s best interest to provide this functionality? Someday, it probably will be, as people get more sophisticated / are more accountable. But we’re probably a few years away from there…
Hi Jim,
this is an awesome analysis !
Thanks
K