Control Groups in Small Populations

Analytical Culture, DataBase Marketing, Driller Q & A, Newsletters, Web Analytics

Jim answers questions from fellow Drillers
(More questions with answers here, Work Overview here, Index of concepts here)

Q: Thank you for your recent article about Control Groups.  Our organization launched an online distance learning program this past August, and I’ve just completed some student behavior analysis for this past semester.

Using weekly RF-Scores based on Recently and Frequently they’ve logged in to courses within the previous three weeks, I’m able to assess their “Risk Level”– how likely they are to stop using the program.  We had a percentage who discontinued the program, but in retrospect, their login behavior and changes in their login behavior gave strong indication they were having trouble before they completely stopped using it.

A: Fantastic!  I have spoken with numerous online educators about this application of Recency – Frequency modeling, as well online research subscriptions, a similar behavioral model.  All reported great results predicting student / subscriber defection rates.

Q: I’m preparing to propose a program for the upcoming semester where we contact students by email and / or phone when their login behavior gives indication that they’re having trouble.  My hope is that by proactively contacting these students, we can resolve issues or provide assistance before things escalate to the point they defect completely.

A: Absolutely, the yield (% students / revenue retained) on a project like this should be excellent.  Plus, you will end up learning a lot about “why”, which will lead to better executions of the “potential dropout” program the more you test it.

Continue reading Control Groups in Small Populations →

Acting on Buyer Engagement

Customer Experience, DataBase Marketing, Measuring Engagement, Web Analytics

Over the years I’ve argued that there is a single, easy to track metric for buyer engagement – Recency.  Though you can develop really complex models for purchase likelihood, just knowing “weeks since last purchase” gets you a long way to understanding how to optimize Marketing and Service programs for profit.

Which brings me to the latest Marketing Science article I have reviewed for the Web Analytics Association, Dynamic Customer Management and the Value of One-to-One Marketing, where the researchers find “customized promotions yield large increases in revenue and profits relative to uniform promotion policies”.  And what variable is most effective when customizing promotions?

The researchers took 56 weeks of purchase behavior from an online store, and used the first 50 weeks to construct a predictive model of purchase behavior.   Inputs to the model included Price, presence of Banner Ads, 3 types of promotions, order sizes, number of orders, merchandise category, demographics, and weeks since last purchase (Recency).

The last 6 weeks of data were used to test the predictive power of the model, and the answer to which variable is most predictive of purchase is displayed in the chart below, click to enlarge:

Weeks since last purchase dominated the predictive power of the model, controlling not only the Natural purchase rate (labeled Baseline in chart above, people who received no promotions) but the response to all three different types of promotion.

Continue reading Acting on Buyer Engagement →

Choosing the Size of Control Groups

DataBase Marketing, Driller Q & A, Newsletters, Web Analytics

Jim answers questions from fellow Drillers
(More questions with answers here, Work Overview here, Index of concepts here)

Q: I am a big fan of your web site and read your Drilling Down book. Great work!

A: Thanks for the kind words!

Q: I was wondering if you could help me picking the right control group size for a project of ours? The population is 25 million telco customers that for which we want to do a long term impact analysis (month by month) in regards to revenue increase versus control group. The marketing initiatives are mix of retention, lifecycle and tactical/seasonal activities. We want to measure revenue increase through any of the marketing activities compared to control group.

A: Great project, this is the kind of idea that can really improve margins if you can find out which specific tactics drop the most profit to the bottom line.

Q:I have searched the web for some help and found calculators that say: On 25 million and smallest expected uplift of 0.1% and highest likely rate of > 5% the calculator gives 250k (1%).  Is that sufficient to calculate the net impact on the remaining base?  Would be very grateful if you could give me your thoughts.

A  Well, it could be and might not be…

Continue reading Choosing the Size of Control Groups →