Poison Control

This post is part of a series on control groups.  The first post is here, a list of all posts in the series here

Using control groups standardizes success tracking across:


so that you begin to really understand what types of marketing create the most value.  There’s only a couple of things left you need to know to start using this gold standard of customer campaign measurement.

I would be remiss if I didn’t at least warn you once to make sure you use a true random sample of the campaign population for the control group.  The direct marketing road is littered with the bodies of those who failed to create a truly random control group for one reason or another, usually accidently, sometimes intentionally. 

For example, they sort by customer number lowest to highest then truncate sample selection before the whole population has been sampled, not realizing the lower the customer number, the longer the person has been a customer.  This creates a bias in control towards “older” customers and screws up the result.  Another common mistake is while trying to make sure the sample is random from a demographic perspective, they end up with a behavioral bias like a higher percentage of Recent buyers in Control than in Test.  There’s nothing that will make your campaign look like it sucked more than stacking Control with customers more likely to respond than those in Test!

The final issue I’d like to bring up is the “organizational stamina” required to execute a controlled testing program. 

In large organizations, a challenge you may encounter is having other people’s campaigns “poison” your control or test groups.  The whole idea of the control is to have this group different in only one way from the test group – they don’t receive your campaign. 

What can happen is someone working with a different segmentation scheme can end up targeting portions of your test or control group, and now you don’t have a controlled test anymore – the control or test has been “poisoned”.

Just to be clear, if the test and control groups are targeted equally, then your test should still be valid, though the overall outcome might be different.  For example, let’s say you have your test and control groups and the company decides to drop a newsletter or announcement to all customers.  Since both test and control will be exposed equally to this newsletter, the incremental effects of your campaign should be preserved. 

Likewise if a national TV campaign is launched.  Your campaign might perform better overall because of the TV, but the lift you get in test versus control should be the same because the TV should affect both test and control equally.

In large organizations where many different groups access the same customer or prospect database, you can see how this poisoning of controlled tests would get to be a mess in quick order.  Without coordination, people would be stomping all over the tests by targeting a piece of a control here and a piece of test there. 

In orgs that are serious about Marketing Productivity, you do typically see a gatekeeper of some kind at the database, making sure that new list pulls do not interfere with any controlled tests that are running.  And yes, sometimes you have to wait to execute your test because there simply are not enough names to go around for the segment you want.  But this is a small price to pay compared to the total chaos of not ever knowing which marketing really works and which does not.

Clearly, there are some Marketing folks who don’t care to know how a campaign really works; “response” is just fine.  In fact, marketing chaos in the database is good for these folks.  Chaos is a fantastic barrier to accountability and the Marketers can just claim ignorance of this control group issue.  That is, until someone with a background in Business Intelligence asks why controls are not being used – and that will not be a pretty day for the Marketer.

But for the analysts out there, I really think it is your duty to start looking at the use of control groups.  Try it a few times and see what you get.  I guarantee you’ll be surprised, and the data you see will open the door to new kinds of thinking and more effective marketing programs for your customer base.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.