4 thoughts on “Control Group Series”

1. Jim, a huge thanks for this series. *Very* Useful and appreciated. Finally taken the time to read and absorb. I suspect I’ll be re-reading a few times…

   Two questions arise from it:
   1. RE: Poisoning. In an online world it is dead-set easy for emails to be forwarded around. Even to people standing around a monitor checking out deals and the like.
   How does this get accounted for? Can it? Would we assume it simply falls into the fuzziness of statistical error? Or take it more seriously? ie We have a inbuilt bias towards poisoning in an online world.
   Recognising that perfect accuracy is impossible.

   2. In the 1st posting in the series you mentioned:
   “… you do segment best customers out for different treatment, don’t you?”

   Why? In what way/basis would you treat them differently? In terms of the analysis? Or in terms of how you treat/entreat them? Apologies if you’ve written to this earlier and I’ve missed it.

   Cheers! and Again my thanks!
   – Steve

2. Steve:

   1. Live with it, it’s a reality and nothing is perfect when it comes to human behavior. One way to detect it is to look for coupon redemptions or discount offers made to the test group happening in the control group; if you don’t see them, then the poisoning “rate” must be low.

   At the same time, call center folks must be instructed on how to handle any “how come she got a discount offer and I didn’t” calls / e-mails with respect. Typically, it would be something like “must have been lost in the mail / spam filter” and then give them the same discount but with a different code so you can track this activity as well.

   However, none of this changes the true financials of the promotion: if sharing goes on and causes control group sales inflation, then it does, and you shouldn’t try to “back out” members of the control group who have taken advantage of the offer.  This just reflects reality.

   2. Because of LifeCycle issues; they should be treated differently if you want to maximize their profitability over the long run. That’s what the messaging series is about:

   http://blog.jimnovo.com/2007/11/25/behavioral-messaging/

   http://blog.jimnovo.com/2008/01/20/messaging-apathetic/

   Pay close attention to the issue of “would have bought anyway”. For best customers who are also Recent (most likely to purchase again), you have to look out for subsidy costs – margin you give away through discounting that you did not have to give away. In an interactive environment, this is a huge (and yet to be widely discovered) issue discussed in this series.

3. 1. Thought so, but went for the Q. just in case. :-)
   The call-centre side is an interesting twist, and the tracking thereof.
   2. Ahh yes. I did read those, I now see the connection far more clearly.
   The point of not giving away margin you don’t have to is just so foreign to my world view/experience. But makes perfect sense!

   *HUGE* thanks!
   – Steve

4. Steve –

   Look, I realize that this stuff is on the advanced side. and for some folks, they really can’t act on any of this right now due to history. It would just be too disruptive to come out and say “By the way, we’re giving away margin that we don’t have to”.

   That said, it would make for great hero work in your next job to come in and ask, “You don’t use control groups?” and proceed to unravel the history there. That’s why I suggest messing around with it to the extent you can now, an underground testing program if you will, so you know what you’re doing when you get the chance to strut your work.

   To start, just exclude a very small group (small enough so nobody will notice if sales drop) of your best, most Recent customers from a couple of consecutive e-mails, and compare their sales, visits, or other actions for the time period to those who get the e-mails.

   You will either find:

   1. Your e-mail program is much more effective than you thought, and generates much more value than you ever imagined. Sales from the excluded group indeed drop and the difference in sales is larger than what you take credit for using “response” as a measurement.

   2. Your e-mail program is much less effective than you thought, and generates little value or negative value when you take into account any discounts. For example, the customers in control have the same sales production as those receiving the e-mails, but without any discounts

   Either way, it’s worth knowing…and the messaging challenge becomes a lot more interesting to play with, as in Kiss, Date, Bribe according to where the customer is in the LifeCycle (how Recent they are)

This site uses Akismet to reduce spam. Learn how your comment data is processed.