Marketing Productivity Blog

This post is part of a series on control groups.  The first post is here, a list of all posts in the series here. 

There are a couple of analytical culture issues I’d like to touch on with using control groups.  Control groups are the gold standard in customer marketing campaign measurement, and at some point, you will be asked to use them.  Heck, you might even get fired for not using them (think new boss comes in), as is the case at Harrahs.

Despite all this, the most obvious stumbling block is you will take a small hit on the revenue line because you’re not dropping the campaign to the control group.  I can hear it now, “But Jim, I can’t afford to take a hit on the revenue”. 

My answer to this is always the same, “You can’t afford not to take the hit, because you absolutely do not know what your true revenue generation is.”  Imagine being in the position of dramatically understating or overstating the true incremental revenue generated by your campaigns – sometimes for years and years.  This is not a pretty picture when it has to be explained.  Personally, I like to avoid that kind of thing!

So I’m just saying, you might want to mess around with control groups a bit before using them gets forced on you.  Controls are a “best practice”, and I don’t know of anyone that can really defend not using best practices.  If your company has a BI group, it’s only a matter of time before somebody over there forces the use of controls.

So how do you deal with the revenue hit?  Like much of analytics, it’s all about explaining what you are doing and why.  Instead of “gross sales”, the campaign focus becomes “sales per customer” – customer centric, if you will.  You are moving to a more customer-focused measurement system.  The goal is lift, improvement in performance, Marketing Productivity.  The tiny loss in sales from the control group is simply a cost of measuring customer marketing properly. 

And trust me, the insights you will get from using controls will be mind blowing.  You will begin to really understand customer behavior, and that’s the first step to creating truly game-changing customer marketing campaigns.

For example, often the increase in sales attribution to your campaigns from using controls will dwarf the loss in sales by not marketing to controls by a factor of 10 or more.  So while you are worrying about dropping half a percentage in campaign revenue by not using a control, you are leaving an increase of 5% in corrected revenue attribution on the table.

How’s that math working for ya?

Yes, this change will probably will be about as painful as explaining to management why you are moving from measuring hits to measuring page views, but that’s life in analytics.  When there is a better way to measure something, you should embrace it – and teach those around you why it makes more sense to measure that way.

More on the cultural issues of using control groups in the next post.

What about you?  Have you faced this “revenue drop” issue with control groups?  How did you handle it?

(This post is more or less an narrative from my presentation at the 2007 Washington D.C. eMetrics / Marketing Optimization Summit)

“You know that campaign with the best response rate ever, the one with $5 million in sales?  We lost over $1 million dollars on it, according to Finance.  Something about the difference between Measuring Campaigns and Measuring Customers.”

- Me, giving my boss at HSN a piece of good news, 1991

That, my friends, was the first time I found out just how important control groups are to measuring the success of customer campaigns in an interactive, always on environment. 

The Finance department – through the Business Intelligence unit – was measuring the net profitability of the campaign at the customer level.  We (Marketing) were measuring the net profitability at the campaign level – based on response to the campaign.  The difference was close to $3 million dollars – from a $1.9 million profit using Marketing’s campaign measurement to a nearly $1 million loss using Finance / BI’s customer measurement.

The crux of this difference is always on, self-service demand, or what Kevin calls Organic Demand.  The only way to measure these customer demand effects accurately – and so the true profitability of campaigns - is with control groups.  Online, this issue is primarily relevant to e-mail marketers (customer marketing) but comes into play in lots of different ways – especially so if you have PPC or display advertising taking credit for generating sales from existing customers.

Seems like there is a lot of confusion around what control groups are and why you should care about them, and I’m hoping this post helps to clear some of that up!  But before I lose you in the details, here is why you should care about this topic:

1.  Tactically: First and foremost, if you’re not using control groups, you are most likely chronically underestimating the sales / visits / whatever KPI you generate.  “Response” is almost always lower than actual demand, because your campaigns generate sales / vists / whatever KPI you cannot track through campaign response mechanisms.  Is full credit for what you contribute to the bottom line important to you?  If so, stick around and read the rest of the post.

2.  Strategically: In a multi-platform, multi-channel, multi-source world, control groups are the gold standard in customer campaign measurement.  You will eventually be required to have a common success measurement that can be used for any situation, as opposed to success measurements “customized” for the quirks of every marketing situation that develops.

If you are not using controls, then your campaign results are always suspect.  The fact nobody has asked you yet to prove the sales you claim to generate are actually generated by your campaigns is not an excuse; that day will come.  Will you be ready?  When “prove it” is on the table, the folks using control groups win over those who are not using them every time.

3. Culturally: The concept of “variance reporting” fundamental to the control group idea is very well understood by senior management.  In fact, despite sounding complex, the control group idea is absolutely the easiest to explain to management and generates a tremendous level of confidence in what you are doing. 

This is why confidence in controlled results is so high: there are no “caveats” and no need for specialized understanding from management of different channels or technologies.  No explanations required for technological causes of error – why does this system say sales were this and this other system say sales were that?  No doubts about the source of the ROI, no questions about external effects.  Clean and simple, elegant in execution.

Interested?  OK, here we go.  Here is the idea in a nutshell.

Let’s talk a little about the idea of “incremental”, as in incremental sales or visits.  Incremental means “extra beyond normal” or what is often called “lift” in the database marketing / BI world.  The central issue is this:  if I spend money on a campaign, I want the campaign to generate incremental sales beyond what I would get if I did not do the campaign.  That’s logical, right?  Why else spend the money, if the campaign is not going to lift my sales over and above what they would have been without the campaign?

In offline retail, Wall Street is always after one KPI – called the “comps”, short for ”same store sales comparisons”.  What they want to know is for stores open at least a year, what were the sales this quarter versus same quarter last year?  That growth, or lift, is what determines how well the company is doing.  The reason is simple: if they just look at gross demand, it can be inflated by opening new stores.  These new store openings mask the true productivity of the operation, and Wall Street knows productivity is what drives profit growth in retail.  So they want to know the incremental sales versus last year of a finite set of stores open at least a year – not the sales of all stores.  In using this approach, they are controlling for the new store openings – removing the influence of them.

And that’s exactly what control groups are for – to remove the influence of any number of factors, and arrive at the true driver of the incremental change.  

When testing the effectiveness of drugs, one of the control groups is often the placebo – the people who take a sugar pill instead of the real drug.  This is done because of the placebo effect – the tendency of a person to feel better when they are taking a drug.  Why is this done?  Because the testers want to measure the real contribution of the drug – the incremental effects over and above the placebo effect.

OK?  So here is how it works in customer marketing:

1.  Choose a population to target with a campaign

2.  Take out a random sample of that population to use as control – the “control group”.  The remaining members of the population after the sample is taken out are called the “test group”.

3.  Send the campaign to the test group, and do nothing to the control group.  Measure the performance of the test versus control over time, and calculate the incremental impact on the test group of receiving the campaign.

A typical email campaign to best customers might look something like this.  Let’s say the campaign has an end date of 1 week after the drop; the customer has to react within a week to take advantage of the offer:

Respectable results for a best customer target – you do segment best customers out for different treatment, don’t you?

Here is what the same campaign probably looks like using a control group, after one week of response:

Note that 10% of targets were taken out as control; the remaining 90,000 received the campaign.

If this campaign had dropped to the entire population of 100,000, the campaign that generated $220,000 in sales really generated only $20,000 in sales, because the incremental sales impact of the campaign was only $20,000 ($.20 per e-mail) versus the control group who received no campaign.  The other $200,000 would have been generated by this customer segment without the campaign.  Follow?

Now at this point, you’re probably saying, “Hey Jim, I get it and all that but there’s no fricking way I’m going to implement this at my current job, I mean, I can’t take a hit like that in performance!” 

To which I would say:

1.  Don’t use controls until you change jobs – you’ll look like a major scientific testing hero at your next job!

2.  You don’t have all the data to make this call yet…we need to talk about what I call “halo effects”.

Halo effects are generally the unintended actions taken by the targets of the campaign.  At a basic level, it’s sales generated because of the campaign that you can’t track back to the campaign using a “campaign response” methodology.

Here’s what this campaign looks like after 6 weeks, when probably almost all the the halo effects would be included.  The numbers for each week are cumulative, they include the sales from the prior weeks:

Now that’s more like it!  If this campaign dropped to the entire population (including the control), it would have generated $295,000.

In this case, there were $75,000 in sales over and above what a “response” measurement of $220,000 shows.  These sales are coming primarily from people who did not respond to the campaign in a way you could track, but did respond to the campaign. 

We’ll dive deeper into explaining how and why this happens, plus address some of the execution and cultural aspects of using control groups in the next post.

Until then, Questions, Comments, Clarifications?

Poor BI; still struggling with broader adoption – as outlined by Ron in the post Four BS BI Trends (And One Good One).  So Gartner identifies BI as the “number one technology issue for 2007″ then immediately pulls out this old chestnut as BI Trend #1: There’s so much data, but too little insight.

Sigh.  

Then I get this comment by Ron Patiro asking: Besides simply not being actionable, what are some of the common pitfalls and tangles of metrics that analysts get themselves into in the pursuit of engagement?

These two ideas are closely related.  The “common pitfalls and tangles of metrics” are often the reason people get a ”so much data, but too little insight” experience.  Let’s explore these issues a bit.

The primary reason you get a “so much data, but too little insight” situation – if you have an analyst to work with the data – is indeed the actionable analysis problem, as Ron P.  points out.  But, there are at least 3 versions of the actionable analysis problem, one obvious and two not so obvious:

• Producing analysis that isn’t actionable at all
• Producing analysis that is valid but too complex to be actionable, and
• Failing to act correctly on a valid and easy to understand analysis 

And often, I find the Root Cause of these three problems (to answer Ron P’s question) to be faulty segmentation logic.  This condition in turn often is born of a situation many web analysts are familiar with by now: No Clear Objective.  But let’s leave the segmentation discussion for later and examine each of three cases above.

One cause of the “too much data, no insight” experience is producing analysis that isn’t actionable at all; it’s literally worthless and cannot be acted upon.  This is the most common vision of the actionable analysis problem – but probably not the one causing the majority of the negative outcomes.  Analysis can be “actionable” from the analysts’ perspective, but not the business perspective.  And if no actual business action takes place, no real insight is gained.

In my experience, people spend an incredible amount of time analyzing things that will never create impact.  Even if the analysis produces something that looks actionable, often the execution is impractical or financially irrelevant and so is not acted upon.  Just because you can “find a pattern” does not mean the business can do anything productive with that pattern.  Randomly “mining for gold” is one of the biggest time wasters around, and why people are often dissatisfied with the result they get from black box data mining projects.  You have to start with an actual business problem of some kind, preferably one that if solved, will increase sales or reduce costs, or no action will be taken.  Otherwise, you have simply created more data to add to the “too much data” side of the problem.

The bottom line for this slice of the problem: The intent and result of the analysis might be actionable, but unless there is a clear business case for acting, you have just contributed to the actionable analysis problem.  In other words, there is a difference between an analysis being “actionable” and having people actually act on it.

The 2nd slice of the “too much data, no insight” problem occurs when the analysis is too complex.   In Marketing at least, complexity introduces error, and probably more importantly, hinders the explanation of the analysis to people who might take action and gain insight.  If a Marketing person can’t understand the analysis, how are they going to formulate a campaign or program to address the problem, never mind get budget to act on the analysis?  Please note I’m talking about the analysis, not solving the problem itself.  Often, an elegantly simple analysis uncovers a problem that will be quite complex to solve.  These are two different issues. 

In fact, I would go as far as to say the more complex the problem is to be solved, the more elegantly simple the analysis needs to be.  The reason is this: the most complex Marketing / Customer problems are usually cross-functional in nature, and to drive success in a cross-functional project, you need rock-simple analysis that galvanizes the team without a lot of second-guessing on the value of a successful outcome.

The bottom line for this slice of the problem: An analysis might be correct and even actionable, but too complex to be acted on.  Complexity opens the analysis up to (often accurate) disbelief in the conclusion, action never takes place, so insight is lost.

The 3rd “too much data, no insight” problem is failure to translate a valid and easy to understand analysis into the correct action.  Here, we are finally moving out of the analytics side of the problem (delivering actionable analysis) and into the Business side.

Why is there failure to act correctly?  I’d submit to you it goes back to the Deconstruction of Marketing – most marketing folks simply don’t understand what to do with “people” as opposed to “Reach and Frequency”.  In other words, they can’t conceptualize how to act successfully against the individual or behavioral segment level as opposed to the nameless, faceless demographic level. 

In my opinion, this is the primary reason why demographics are so overused in customer analysis, especially online – the marketing folks simply can’t get out of that box, it’s where the “actionability” starts for them.  The problem with this thought process, as has been pointed out, is that demographics often have little to do with behavior.  Behavior predicts behavior; demographics are mostly coincidental.  Yet the analyst, looking to produce a successful project, often will allow themselves to be dragged into endless demographic segmentation that is primarily a waste of time (unless you are a media site and sell demos) and leads to false conclusions, which lead to failed or inconsistent implementation.

The bottom line for this slice of the problem: the analysis identified a problem or opportunity, but in the end, the execution against the analysis was flawed and ultimately delivered poor or no real insight.  By the way, I think this third form of failure to deliver insight is the most common – much more common than most people think.  Why?  It’s the hidden one, the one that’s not so obvious and much easier to push under the table.

So there you have it.  Three versions of the “actionable analysis” problem that lead directly to the “so much data, but too little insight” issue.  I think #3 is probably the most prevalent; a lot of analysis “fails” not because of poor analysis, but poor execution against the analysis.

What do you think?  Have you delivered a clearly actionable analysis, one that is capable of real business impact, only to have the execution against the analysis botched?

Perhaps more importantly, were you able to do anything about the botched execution?  Were you able to turn it around?  How did you make that happen?

Or, is execution not really your problem – if Marketing (or whoever) screws it up, then they screw it up?

Interesting article in MultiChannel Merchant about sourcing sales across catalog and online using fractional allocation models.  I’m pretty sure “allocation” and “attribution” are really different concepts, though they seem to be used interchangeably right now.  Let’s just say from reading the article allocation sounds more like a gut feel thing and attribution, from my experience, implies the use of a mathematical model of some kind. 

I know Kevin rails against a lot of the so-called matchback analysis done in catalog and I have to agree with him; that practice is a whole lot more like allocation then attribution in my book, particularly when it is pretty easy to measure the real source from a lift in demand perspective by using control groups.  Take a random sample of the catalog target group, exclude it from the mailing, and compare the purchase behavior in this group with those customers who get the catalog over some time period.  That should give you an idea of what the incremental demand from catalog is - just look at gross sales per customer.  We did this at HSN for every promotion, since the TV was “always on” and creating demand by itself. 

So does a web site. 

Just because someone was mailed a catalog and then at some point later on ordered from a web site does not mean they ordered because they received the catalog; heck, you don’t even know for sure if they even received the catalog – as anyone who has used seeded lists knows.  And just because someone was exposed to an ad online doesn’t mean the ad had anything to do with a subsequent online order – even if you believe in view-through.

Anyway, I see lots of people doing what I would call allocation rather than attribution in the web analytics space, and when Jacques Warren asked me about this topic the other day, I decided it might make a good post. 

You have to understand this same discussion has been going on at least 25 years already in offline, so there is a ton of history in terms of best practices and real experience behind the approach many folks favor.  And there is a twist to the online version I don’t think many folks are considering.  So for what it’s worth, here’s my take…

For most folks, the simplest and most reliable way to attribute demand is to choose either first campaign or last campaign and stick to it.  The words simplest and reliable were chosen very specifically.  For the very few folks who have the right people, the right tools, and the right data, it is possible to build mathematically precise attribution models.  The word precise was also chosen specifically.   I will go into more detail on these choices below after some background.

Choosing first or last campaign for attribution is not ignoring the effects of other campaigns, but simply recognizes you cannot measure these effects accurately, and to create any “allocation model” will be an exercise in navel gazing.

Unfortunately, a lot of this kind of thing goes on in web analytics – instead of admitting something can’t be measured accurately, folks substitute a “model” which is worse than admitting the accuracy problem, because now you are saying you have a “measurement” when you don’t.  People sit around with a web analytics report, and say, “Well, the visitor saw the PPC ad, then they did an organic search, then they saw a banner, so we will give 1/3 of the sales credit to each” or worse, “we will allocate the credit for sales based on what we spend on each exposure”.

This approach is worse then having no model at all, because I often see these models used improperly, (for example) to “justify budget” – if you allocate a share of responsibility for outcome to PPC, then you get to keep a budget that would otherwise be “optimized” away.  A similar argument is being made by a few of the folks in the MultiChannel Merchant article above to justify catalog spend. 

This is nuts, in my opinion.

I believe the core analytical culture problem at work here (if you are interested) is this:

Difference between Accuracy and Precision
http://en.wikipedia.org/wiki/Accuracy

I’d argue that given a choice, it’s more important to be precise than accurate – reproducibility is more important (especially to management) than getting the exact number right.  Reproducibility is, after all, at the core of the scientific testing method, isn’t it?  If you can’t repeat the test and get the same results, you don’t have a valid hypothesis. 

And given the data stream web analytics folks are working with – among the dirtiest data around in terms of accuracy – then why would people spend so much time trying to build an “accurate” model?  Better to be precise – always using first campaign or last campaign – than to create the illusion of accuracy with an allocation model that is largely made up from thin air. 

When I make the statement above, I’m excluding a team of Ph.D. level statisticians with the best tools and data scrubbing developing the model, though I suspect only a handful of companies doing these models actually fit that description.  For the vast majority of companies, the principle of Occam’s Razor rules here; what I want is reliability and stability; every time I do X, I get Y – even if I don’t know exactly (accurately) how I get Y from X. 

Ask yourself if that level of accuracy really matters – if every time I put in $1 I get back $3, over and over, does it matter specifically and totally accurately exactly how that happens?

Whether to use first or last campaign is a matter of philosophy / culture and not one of measurement.  If you believe that in general, the visitor / customer mindset is created by exposure or interaction to the first campaign, and that without this favorable context none of the subsequent campaigns would be very effective, then use first campaign.

This is generally my view and the view of many offline direct marketing folks I know.  Here is why.  The real “leverage” in acquisition campaigns is the first campaign – the first campaign has the hardest job, if you will – so if you are going to optimize, the biggest bang for the buck is in optimizing first campaign, where if you get it wrong, all the rest of the campaigns are negatively affected.  This is the “leverage” part of the idea; on any campaign other than first, you can’t make a statement like this.  So it follows that every campaign should be optimized as “first campaign”, since you don’t normally control which campaign will be seen first.

Some believe that the sale or visit would not have occurred if the last campaign was not effective, and all other campaigns are just “prep” for that campaign to be successful.  Perhaps true, but it doesn’t fit my model of the world – unless you know that first campaign sucks.  If you know that, then why wouldn’t you fix it or kill it, for heaven’s sake?

All of the above said, if you have the chops, the data, and the tools, you can produce attribution models that will provide direction on “weighting” the effect of different campaigns.  These ”marketing mix” models are used all the time offline, and are usually the product of high level statistical models.   By the way, they’re not generally accurate, but they are precise.  I do X, I get Y.

You can produce a similar kind of information through very tightly testing using control groups, but that’s not much help for acquisition because you usually can’t get your hands on a good control group.  So for acquisition you are left with trying to synch time periods and doing sequential or layered testing.

For example, in June we are going to kill all the AdSense advertising and see what happens to our AdWords advertising – what happens to impressions, CTR, conversion, etc.  Then in July we will kick AdSense on again and see what happens to the same variables, along with tracking as best we can any overlapping exposures. 

Then given this info, we decide about allocation using the human brain and database marketing experience.

This approach is not accurate, but I’d rather be precise and “directionally right” then accurate and be absolutely wrong, if you know what I mean.  This test approach should give you directional results if executed correctly – the spread for the AdSense OFF / ON test results should be healthy, and you should be able to repeat the test result with some consistency.

Bottom line - it doesn’t really matter exactly what is happening, does it?  Do you need an accurate accounting of the individual effects of each campaign in a multiple campaign sequence?  No.  What you need is a precise (reliable and reproducible) way to understand the final outcome of the marketing mix.

Even if you think you have an accurate accounting of the various campaign contributions, what makes you think you can get that with data as dirty as web data is?  Despite the attempt at accuracy, all you have to do is think through cookies, multiple computers, systems issues, and web architecture itself to understand that after all that work, you still don’t have an accurate result. 

Hopefully it is more precise than simply using first campaign.

Thoughts from you on this topic?  I know there are at least two “marketing mix” folks on the feed…