If you’re looking to measure lift for a single campaign, you should always pull a control group.

If you’re looking to measure lift for a series of campaigns to the same targets, you can pull one control group for the series, and optionally for each campaign in the series.

If you’re measuring very long term effects, for example, the first 5 years of a loyalty program, then you’d want to use the same control group for the entire span and perhaps for the life of the program – but this control is really only valid for the original sample it was taken from. There needs to be a realization the validity of this control for the entire customer population tends to expire over time as the “age” of customers in it increases.

You could argue a control taken in Year 1 was still valid in Year 3, but strictly speaking, it’s only valid against the set of Year 1 customers it was pulled from. So while you could still compare customers of the Year 1 vintage to this Year 1 control, if you have a 40% annual attrition rate, by Year 3 this Year 1 vintage is becoming a pretty small percentage of the overall customer population.

Perhaps you can assume that Year 2 and Year 3 customers “are the same” as Year 1, but chances are they are not – the Year 1 customers tend to be early adopters and so have a different profile – especially in a pure online kind of environment.

So to be sure of what you’re getting for lift, I would create new controls every year or 2 and track them all. This approach not only checks for variations in customer quality and program effectiveness for every year of the program, but also allows a prediction of the program value based on any changes in customer mix.

Make sense? Thanks for the question, a very good one!

Update: Whoops! Looks like Nick had it covered already…

This is a fairly common concept. There’s not really an agreed name for such long-term controls, but I tend to call them fallows.

Having a fallow group is useful if what you want do is to understand the impact of an entire marketing program. It’s particularly useful if you introduce a new way of doing things and want to compare it to business as usual, or if you just want to prove that marketing has a positive impact (assuming it does…)

However, it is NOT a substitute for a regular campaign control group. The whole idea of campaign controls is that the ONLY systematic difference between the group in the campaign and the controls is the campaign. If you try to use fallows as a campaign control, you can’t isolate the impact of your particular campaign, so you can’t say what’s due to that campaign and what’s due to all the other stuff they missed out on.

Does that help?

Regards

Nick

Please do NOT use my real name or company.

Actually, if I have time, I’ll do the (sort-of) equivalent Windows one today or tomorrow.

I’ve been holding off on reading this series of yours Jim, to when I can devote the time to really understand. After a quick skim of some of Nick’s cartoons, I’m regretting the earlier loss to otherwise increasing my “laughing out loud” quotient. :-)

Cheers! (and thanks!)
- Steve

I think the cartoons are working at least a bit as an educational tool. They definitely help people outside the industry to understand what I do, but I think they also help people who know more to keep in mind key distinctions,and if people have read them you can kind-of refer back to them to make a point. Time will tell.

…And I just love the idea that as I learn more I might get to find them funnier :-)

Of course, the paradox of the cartoons as a learning tool is the more you know about this topic, the funnier the cartoons are! Have you had any luck with them from an educational perspective?

The cartoons, as you may have guessed, evolved from my own frustration at how subtle some of these concepts are, so now I’m trying to break them down into one-concept-per-four-box-cartoon chunks. And it’s still hard!

The quote I always use about control groups comes from Robert McNamara, of all people, (who was talking, of course, in a different context): “We have to find a way of making the important measurable, instead of making the measurable important.” That’s control groups.

I think the case you describe is not rare, I think it happens all the time with best customers. It’s just that people don’t segment out best customers and use controls for them so they can’t see this happen.

Frankly, I was trying to stay away from this case because:

1. I have covered it several times before
2. The complexity of explaining it to people

in favor of just trying to get some folks interested in using control groups.

Speaking of difficult concepts to explain, the notion “lack of marketing” can actually increase sales is another difficult one to swallow for most folks, but I’ve seen it many times before. Thanks for stopping in and I’ll be following your blog…By the way, those cartoons are a riot!

http://scientificmarketer.com/2007/09/controls.html

OK, so I admit that such cases are rare. But it’s not at all rare for targeting using response models to direct mailings to people who would have bought anyway, and if the mailing is incentivized this again can lead to a real drop in revenue.

Using control groups really should be a complete no brainer.