This post is an index for the Control Group series. The following posts were written sequentially but appear on the blog in reverse chronological order which makes a hell of a mess of trying to understand a somewhat complicated topic. So instead, try reading them sequentially using this index:
Poison Control
This post is part of a series on control groups. The first post is here, a list of all posts in the series here.
Using control groups standardizes success tracking across:
Platforms
Sources
Channels
so that you begin to really understand what types of marketing create the most value. There’s only a couple of things left you need to know to start using this gold standard of customer campaign measurement.
I would be remiss if I didn’t at least warn you once to make sure you use a true random sample of the campaign population for the control group. The direct marketing road is littered with the bodies of those who failed to create a truly random control group for one reason or another, usually accidently, sometimes intentionally.
For example, they sort by customer number lowest to highest then truncate sample selection before the whole population has been sampled, not realizing the lower the customer number, the longer the person has been a customer. This creates a bias in control towards “older” customers and screws up the result. Another common mistake is while trying to make sure the sample is random from a demographic perspective, they end up with a behavioral bias like a higher percentage of Recent buyers in Control than in Test. There’s nothing that will make your campaign look like it sucked more than stacking Control with customers more likely to respond than those in Test!
The final issue I’d like to bring up is the “organizational stamina” required to execute a controlled testing program.
In large organizations, a challenge you may encounter is having other people’s campaigns “poison” your control or test groups. The whole idea of the control is to have this group different in only one way from the test group – they don’t receive your campaign.
What can happen is someone working with a different segmentation scheme can end up targeting portions of your test or control group, and now you don’t have a controlled test anymore – the control or test has been “poisoned”.
Just to be clear, if the test and control groups are targeted equally, then your test should still be valid, though the overall outcome might be different. For example, let’s say you have your test and control groups and the company decides to drop a newsletter or announcement to all customers. Since both test and control will be exposed equally to this newsletter, the incremental effects of your campaign should be preserved.
Likewise if a national TV campaign is launched. Your campaign might perform better overall because of the TV, but the lift you get in test versus control should be the same because the TV should affect both test and control equally.
In large organizations where many different groups access the same customer or prospect database, you can see how this poisoning of controlled tests would get to be a mess in quick order. Without coordination, people would be stomping all over the tests by targeting a piece of a control here and a piece of test there.
In orgs that are serious about Marketing Productivity, you do typically see a gatekeeper of some kind at the database, making sure that new list pulls do not interfere with any controlled tests that are running. And yes, sometimes you have to wait to execute your test because there simply are not enough names to go around for the segment you want. But this is a small price to pay compared to the total chaos of not ever knowing which marketing really works and which does not.
Clearly, there are some Marketing folks who don’t care to know how a campaign really works; “response” is just fine. In fact, marketing chaos in the database is good for these folks. Chaos is a fantastic barrier to accountability and the Marketers can just claim ignorance of this control group issue. That is, until someone with a background in Business Intelligence asks why controls are not being used – and that will not be a pretty day for the Marketer.
But for the analysts out there, I really think it is your duty to start looking at the use of control groups. Try it a few times and see what you get. I guarantee you’ll be surprised, and the data you see will open the door to new kinds of thinking and more effective marketing programs for your customer base.
Are You in Control?
This post is part of a series on control groups. The first post is here, a list of all posts in the series here.
Mike Moran recently wrote about how Search Marketing is Direct Marketing. I myself commented”the Web is a direct marketing machine” back in 2001 when most people hated the idea of PPC marketing and thought it would never catch on.
Most of the critical breakthroughs in optimizing online marketing have been based on direct or database marketing principles that have been around for decades. In my last post on Control Groups, I said “the insights you will get from using controls will be mind blowing. You will begin to really understand customer behavior, and that’s the first step to creating truly game-changing customer marketing campaigns”.
I have some examples for you.
Check out this list detailing some of those insights. Sure, they are in the form of “mistakes” but they are insights nonetheless. See 41 Timeless Ways to Screw Up Direct Marketing by Nicholas J. Radcliffe.
The interesting thing about this list is most of these mistakes can only be identified if you are using control groups; that’s how important the concept is to customer-centric marketing. For some mistakes on this list, you will think to yourself, “How could they ever measure that?”
The answer is one you are familiar with: repeated testing, in this case over many different industries and using many different data sets. But you have to add controls to the test or you won’t see the effects.
Many of these mistakes are things you hear the CRM / customer-centric / CGM pundits talk about all the time, stuff like talking down to the customer, over-communicating, or being intrusive. But these same folks never offer any conclusive proof of the financial damage these acts can cause; it’s all “gut feel”.
How would you like to be able to prove what the damage caused by reckless marketing is really worth?
Online marketers are currently making many of these same 41 mistakes – they just don’t know it yet. #17 and #19 are going to be very disruptive when they become widely understood. If you want to understand more about these mistakes, a specific example is here or for a broader framework to work from, see here.
But the real question at hand is this: Will you be a driver of the next level of achievement in online customer marketing by suggesting (and eventually requiring) the use of Control Groups?
In the final post of this series, we’ll touch on two challenges with the implementation of control groups.