Category Archives: Web Analytics

Are You in Control?

Analytical Culture, DataBase Marketing, Web Analytics

This post is part of a series on control groups. The first post is here, a list of all posts in the series here.

Mike Moran recently wrote about how Search Marketing is Direct Marketing. I myself commented”the Web is a direct marketing machine” back in 2001 when most people hated the idea of PPC marketing and thought it would never catch on.

Most of the critical breakthroughs in optimizing online marketing have been based on direct or database marketing principles that have been around for decades. In my last post on Control Groups, I said “the insights you will get from using controls will be mind blowing. You will begin to really understand customer behavior, and that’s the first step to creating truly game-changing customer marketing campaigns”.

I have some examples for you.

Check out this list detailing some of those insights. Sure, they are in the form of “mistakes” but they are insights nonetheless. See 41 Timeless Ways to Screw Up Direct Marketing by Nicholas J. Radcliffe.

The interesting thing about this list is most of these mistakes can only be identified if you are using control groups; that’s how important the concept is to customer-centric marketing. For some mistakes on this list, you will think to yourself, “How could they ever measure that?”

The answer is one you are familiar with: repeated testing, in this case over many different industries and using many different data sets. But you have to add controls to the test or you won’t see the effects.

Many of these mistakes are things you hear the CRM / customer-centric / CGM pundits talk about all the time, stuff like talking down to the customer, over-communicating, or being intrusive. But these same folks never offer any conclusive proof of the financial damage these acts can cause; it’s all “gut feel”.

How would you like to be able to prove what the damage caused by reckless marketing is really worth?

Online marketers are currently making many of these same 41 mistakes – they just don’t know it yet. #17 and #19 are going to be very disruptive when they become widely understood. If you want to understand more about these mistakes, a specific example is here or for a broader framework to work from, see here.

But the real question at hand is this: Will you be a driver of the next level of achievement in online customer marketing by suggesting (and eventually requiring) the use of Control Groups?

In the final post of this series, we’ll touch on two challenges with the implementation of control groups.

Culture of Control (Groups)

Analytical Culture, DataBase Marketing, Web Analytics

This post is part of a series on control groups. The first post is here, a list of all posts in the series here.

There are a couple of analytical culture issues I’d like to touch on with using control groups. Control groups are the gold standard in customer marketing campaign measurement, and at some point, you will be asked to use them. Heck, you might even get fired for not using them – think new boss comes in.

Despite all this, the most obvious stumbling block is you will take a small hit on the revenue line because you’re not dropping the campaign to the control group. I can hear it now, “But Jim, I can’t afford to take a hit on the revenue”.

My answer to this is always the same, “You can’t afford not to take the hit, because you absolutely do not know what your true revenue generation is.” Imagine being in the position of dramatically understating or overstating the true incremental revenue generated by your campaigns – sometimes for years and years. This is not a pretty picture when it has to be explained. Personally, I like to avoid that kind of thing!

So I’m just saying, you might want to mess around with control groups a bit before using them gets forced on you. Controls are a “best practice”, and I don’t know of anyone that can really defend not using best practices. If your company has a BI group, it’s only a matter of time before somebody over there forces the use of controls.

So how do you deal with the revenue hit? Like much of analytics, it’s all about explaining what you are doing and why. Instead of “gross sales”, the campaign focus becomes “sales per customer” – customer centric, if you will. You are moving to a more customer-focused measurement system. The goal is lift, improvement in performance, Marketing Productivity. The tiny loss in sales from the control group is simply a cost of measuring customer marketing properly.

And trust me, the insights you will get from using controls will be mind blowing. You will begin to really understand customer behavior, and that’s the first step to creating truly game-changing customer marketing campaigns.

For example, often the increase in sales attribution to your campaigns from using controls will dwarf the loss in sales by not marketing to controls by a factor of 10 or more. So while you are worrying about dropping half a percentage in campaign revenue by not using a control, you are leaving an increase of 5% in corrected revenue attribution on the table.

How’s that math working for ya?

Yes, this change will probably will be about as painful as explaining to management why you are moving from measuring hits to measuring page views, but tha’s life in analytics. When there is a better way to measure something, you should embrace it – and teach those around you why it makes more sense to measure that way.

More on the cultural issues of using control groups in the next post.

What about you? Have you faced this “revenue drop” issue with control groups? How did you handle it?

Control Group Benefits

DataBase Marketing, Web Analytics

This post is part of a series on control groups.  The first post is here, a list of all posts in the series here.

Last time, we finished with a look at halo effects – sales from people who did not respond to the campaign in a way you could track, but did respond to the campaign. Halo effects are the primary reason your campaign measurements probably underestimate the true response you are generating from e-mail campaigns; the only way to measure these “extra sales” is to use control groups.

But why do these sales happen? Basic marketing; you created awareness and it resulted in a sale. For example:

The offer was good for only a week. People wanted to make a purchase, forgot about it, and remembered after the offer expired. So they went to the site directly by typing in the URL and made a purchase without the offer.

This simple story – with probably dozens or hundreds of minor variations – is where the halo effect sales come from.  Your marketing worked, it just did not work the way you thought it would or in a way you could track.

However, if you were measuring success at the Customer level instead of the Campaign level, you would see these sales coming in after the campaign was over by comparing the per customer activity of the test group with the control group.

In other words, Control Groups assign credit to your campaign for:

Brand Execution / Top of Mind

Great Copy / Customized Offer

Timing – right message, at the right time

Engagement Factors

that you can’t measure in any other way. Measuring Customers instead of Campaigns tells you exactly where the Value was created, and enabling accurate Value measurement allows you to correctly attribute and understand Value creation.

In fact, I could argue that Response metrics are really about your selling process, not the customer buying process – response is not a customer-centric measurement approach. Response doesn’t take into account the fact that customers will do things the way they want to, not the way you want them to.

Response is a systems-oriented, mechanical construct that is blind to the effects of emotion. Control Groups are human-oriented, behavioral constructs that are able to include emotion in the measurement of outcome. Not that you will know exactly “how” these halo effects occur – you have to let go of that accuracy thing – but a least you will be consistently and completely measuring “what” happened.

And using Control Groups, you can now Measure the un-Measurable – you don’t need “response” to carry the load. You can measure the ROI of non-response, “feel good” campaigns, for example:

Birthday / anniversary cards – without coupons

“We Love You” / KISS Campaigns

Just calling to see if you need anything

Special Events

These are some of the Highest ROI Best Customer Marketing tactics available. But people don’t use them, because they can’t figure out how to measure the ROI. Measurement is simple with Control Groups, and the Marketing folks will love you for using controls because they can actually get back to doing some real Marketing instead of just creating offers all the time.

That’s boring – for both the Marketers and the customers!

Bullet-proof accountability, simple to explain to management, your friend, the Control Group. Next time we will get into some operational and cultural issues surrounding the use of Control Groups.

Comments? Questions? Have you used control groups, offline or online?  How did it work out for you?