This post is part of a series on control groups.  The first post is here, a list of all posts in the series here. 

Using control groups standardizes success tracking across:

Platforms
Sources
Channels

so that you begin to really understand what types of marketing create the most value.  There’s only a couple of things left you need to know to start using this gold standard of customer campaign measurement.

I would be remiss if I didn’t at least warn you once to make sure you use a true random sample of the campaign population for the control group.  The direct marketing road is littered with the bodies of those who failed to create a truly random control group for one reason or another, usually accidently, sometimes intentionally. 

For example, they sort by customer number lowest to highest then truncate sample selection before the whole population has been sampled, not realizing the lower the customer number, the longer the person has been a customer.  This creates a bias in control towards “older” customers and screws up the result.  Another common mistake is while trying to make sure the sample is random from a demographic perspective, they end up with a behavioral bias like a higher percentage of Recent buyers in Control than in Test.  There’s nothing that will make your campaign look like it sucked more than stacking Control with customers more likely to respond than those in Test!

The final issue I’d like to bring up is the “organizational stamina” required to execute a controlled testing program. 

In large organizations, a challenge you may encounter is having other people’s campaigns “poison” your control or test groups.  The whole idea of the control is to have this group different in only one way from the test group – they don’t receive your campaign. 

What can happen is someone working with a different segmentation scheme can end up targeting portions of your test or control group, and now you don’t have a controlled test anymore – the control or test has been “poisoned”.

Just to be clear, if the test and control groups are targeted equally, then your test should still be valid, though the overall outcome might be different.  For example, let’s say you have your test and control groups and the company decides to drop a newsletter or announcement to all customers.  Since both test and control will be exposed equally to this newsletter, the incremental effects of your campaign should be preserved. 

Likewise if a national TV campaign is launched.  Your campaign might perform better overall because of the TV, but the lift you get in test versus control should be the same because the TV should affect both test and control equally.

In large organizations where many different groups access the same customer or prospect database, you can see how this poisoning of controlled tests would get to be a mess in quick order.  Without coordination, people would be stomping all over the tests by targeting a piece of a control here and a piece of test there. 

In orgs that are serious about Marketing Productivity, you do typically see a gatekeeper of some kind at the database, making sure that new list pulls do not interfere with any controlled tests that are running.  And yes, sometimes you have to wait to execute your test because there simply are not enough names to go around for the segment you want.  But this is a small price to pay compared to the total chaos of not ever knowing which marketing really works and which does not.

Clearly, there are some Marketing folks who don’t care to know how a campaign really works; “response” is just fine.  In fact, marketing chaos in the database is good for these folks.  Chaos is a fantastic barrier to accountability and the Marketers can just claim ignorance of this control group issue.  That is, until someone with a background in Business Intelligence asks why controls are not being used - and that will not be a pretty day for the Marketer.

But for the analysts out there, I really think it is your duty to start looking at the use of control groups.  Try it a few times and see what you get.  I guarantee you’ll be surprised, and the data you see will open the door to new kinds of thinking and more effective marketing programs for your customer base.

This post is part of a series on control groups.  The first post is here, a list of all posts in the series here.

Mike Moran recently wrote about how Search Marketing is Direct Marketing.  I myself commented “the Web is a direct marketing machine” back in 2001 when most people hated the idea of PPC marketing and thought it would never catch on.

Most of the critical breakthroughs in optimizing online marketing have been based on direct or database marketing principles that have been around for decades.  In my last post on Control Groups, I said “the insights you will get from using controls will be mind blowing.  You will begin to really understand customer behavior, and that’s the first step to creating truly game-changing customer marketing campaigns”.

I have some examples for you.

Check out this list detailing some of those insights.  Sure, they are in the form of “mistakes” but they are insights nonetheless.  See 41 Timeless Ways to Screw Up Direct Marketing by Nicholas J. Radcliffe.

The interesting thing about this list is most of these mistakes can only be identified if you are using control groups; that’s how important the concept is to customer-centric marketing. For some mistakes on this list, you will think to yourself, “How could they ever measure that?”

The answer is one you are familiar with: repeated testing, in this case over many different industries and using many different data sets.  But you have to add controls to the test or you won’t see the effects.

Many of these mistakes are things you hear the CRM / customer-centric / CGM pundits talk about all the time, stuff like talking down to the customer, over-communicating, or being intrusive.  But these same folks never offer any conclusive proof of the financial damage these acts can cause; it’s all “gut feel”.

How would you like to be able to prove what the damage caused by reckless marketing is really worth?

Online marketers are currently making many of these same 41 mistakes – they just don’t know it yet.  #17 and #19 are going to be very disruptive when they become widely understood.  If you want to understand more about these mistakes, a specific example is here or for a broader framework to work from, see here.

But the real question at hand is this: Will you be a driver of the next level of achievement in online customer marketing by suggesting (and eventually requiring) the use of Control Groups?

In the final post of this series, we’ll touch on two challenges with the implementation of control groups.

This post is part of a series on control groups.  The first post is here, a list of all posts in the series here.

There are a couple of analytical culture issues I’d like to touch on with using control groups.  Control groups are the gold standard in customer marketing campaign measurement, and at some point, you will be asked to use them.  Heck, you might even get fired for not using them –  think new boss comes in.

Despite all this, the most obvious stumbling block is you will take a small hit on the revenue line because you’re not dropping the campaign to the control group.  I can hear it now, “But Jim, I can’t afford to take a hit on the revenue”.

My answer to this is always the same, “You can’t afford not to take the hit, because you absolutely do not know what your true revenue generation is.”  Imagine being in the position of dramatically understating or overstating the true incremental revenue generated by your campaigns – sometimes for years and years.  This is not a pretty picture when it has to be explained.  Personally, I like to avoid that kind of thing!

So I’m just saying, you might want to mess around with control groups a bit before using them gets forced on you.  Controls are a “best practice”, and I don’t know of anyone that can really defend not using best practices.  If your company has a BI group, it’s only a matter of time before somebody over there forces the use of controls.

So how do you deal with the revenue hit?  Like much of analytics, it’s all about explaining what you are doing and why.  Instead of “gross sales”, the campaign focus becomes “sales per customer” – customer centric, if you will.  You are moving to a more customer-focused measurement system.  The goal is lift, improvement in performance, Marketing Productivity. The tiny loss in sales from the control group is simply a cost of measuring customer marketing properly.

And trust me, the insights you will get from using controls will be mind blowing.  You will begin to really understand customer behavior, and that’s the first step to creating truly game-changing customer marketing campaigns.

For example, often the increase in sales attribution to your campaigns from using controls will dwarf the loss in sales by not marketing to controls by a factor of 10 or more.  So while you are worrying about dropping half a percentage in campaign revenue by not using a control, you are leaving an increase of 5% in corrected revenue attribution on the table.

How’s that math working for ya?

Yes, this change will probably will be about as painful as explaining to management why you are moving from measuring hits to measuring page views, but tha’s life in analytics. When there is a better way to measure something, you should embrace it – and teach those around you why it makes more sense to measure that way.

More on the cultural issues of using control groups in the next post.

What about you?  Have you faced this “revenue drop” issue with control groups?  How did you handle it?