There are a couple of analytical culture issues I’d like to touch on with using control groups. Control groups are the gold standard in customer marketing campaign measurement, and at some point, you will be asked to use them. Heck, you might even get fired for not using them – think new boss comes in.
Despite all this, the most obvious stumbling block is you will take a small hit on the revenue line because you’re not dropping the campaign to the control group. I can hear it now, “But Jim, I can’t afford to take a hit on the revenue”.
My answer to this is always the same, “You can’t afford not to take the hit, because you absolutely do not know what your true revenue generation is.” Imagine being in the position of dramatically understating or overstating the true incremental revenue generated by your campaigns – sometimes for years and years. This is not a pretty picture when it has to be explained. Personally, I like to avoid that kind of thing!
So I’m just saying, you might want to mess around with control groups a bit before using them gets forced on you. Controls are a “best practice”, and I don’t know of anyone that can really defend not using best practices. If your company has a BI group, it’s only a matter of time before somebody over there forces the use of controls.
So how do you deal with the revenue hit? Like much of analytics, it’s all about explaining what you are doing and why. Instead of “gross sales”, the campaign focus becomes “sales per customer” – customer centric, if you will. You are moving to a more customer-focused measurement system. The goal is lift, improvement in performance, Marketing Productivity. The tiny loss in sales from the control group is simply a cost of measuring customer marketing properly.
And trust me, the insights you will get from using controls will be mind blowing. You will begin to really understand customer behavior, and that’s the first step to creating truly game-changing customer marketing campaigns.
For example, often the increase in sales attribution to your campaigns from using controls will dwarf the loss in sales by not marketing to controls by a factor of 10 or more. So while you are worrying about dropping half a percentage in campaign revenue by not using a control, you are leaving an increase of 5% in corrected revenue attribution on the table.
How’s that math working for ya?
Yes, this change will probably will be about as painful as explaining to management why you are moving from measuring hits to measuring page views, but that’s life in analytics. When there is a better way to measure something, you should embrace it – and teach those around you why it makes more sense to measure that way.
More on the cultural issues of using control groups in the next post.
What about you? Have you faced this “revenue drop” issue with control groups? How did you handle it?