Category Archives: Driller Q & A

Control Groups in Small Populations

Analytical Culture, DataBase Marketing, Driller Q & A, Newsletters, Web Analytics

Jim answers questions from fellow Drillers
(More questions with answers here, Work Overview here, Index of concepts here)

Q: Thank you for your recent article about Control Groups.  Our organization launched an online distance learning program this past August, and I’ve just completed some student behavior analysis for this past semester.

Using weekly RF-Scores based on Recently and Frequently they’ve logged in to courses within the previous three weeks, I’m able to assess their “Risk Level”– how likely they are to stop using the program.  We had a percentage who discontinued the program, but in retrospect, their login behavior and changes in their login behavior gave strong indication they were having trouble before they completely stopped using it.

A: Fantastic!  I have spoken with numerous online educators about this application of Recency – Frequency modeling, as well online research subscriptions, a similar behavioral model.  All reported great results predicting student / subscriber defection rates.

Q: I’m preparing to propose a program for the upcoming semester where we contact students by email and / or phone when their login behavior gives indication that they’re having trouble.  My hope is that by proactively contacting these students, we can resolve issues or provide assistance before things escalate to the point they defect completely.

A: Absolutely, the yield (% students / revenue retained) on a project like this should be excellent.  Plus, you will end up learning a lot about “why”, which will lead to better executions of the “potential dropout” program the more you test it.

Continue reading Control Groups in Small Populations →

Choosing the Size of Control Groups

DataBase Marketing, Driller Q & A, Newsletters, Web Analytics

Jim answers questions from fellow Drillers
(More questions with answers here, Work Overview here, Index of concepts here)

Q: I am a big fan of your web site and read your Drilling Down book. Great work!

A: Thanks for the kind words!

Q: I was wondering if you could help me picking the right control group size for a project of ours? The population is 25 million telco customers that for which we want to do a long term impact analysis (month by month) in regards to revenue increase versus control group. The marketing initiatives are mix of retention, lifecycle and tactical/seasonal activities. We want to measure revenue increase through any of the marketing activities compared to control group.

A: Great project, this is the kind of idea that can really improve margins if you can find out which specific tactics drop the most profit to the bottom line.

Q:I have searched the web for some help and found calculators that say: On 25 million and smallest expected uplift of 0.1% and highest likely rate of > 5% the calculator gives 250k (1%).  Is that sufficient to calculate the net impact on the remaining base?  Would be very grateful if you could give me your thoughts.

A  Well, it could be and might not be…

Continue reading Choosing the Size of Control Groups →

Customer Value in the Freemium Model

DataBase Marketing, Driller Q & A, Newsletters, Web Analytics

Jim answers questions from fellow Drillers
(More questions with answers here, Work Overview here, Index of concepts here)

Q: You kindly clarified a few issues when I was reading Drilling Down earlier this year – so I hope you don’t mind the direct email.

A: Yes, I remember!

I am working for www.XYZ.com, a social networking / virtual world site based abroad but visitors are 85% US.

Our growth up to now has been mainly viral and in the summer we hit 1.2M UVs operating on the Freemium model with only 5% of our registered users converting to paying customers and a significant portion of our revenue coming from ads.  On average our customers are active on the site for something like 4 months making their first purchase around day 28. 

But to take us to the next stage we are embarking on some marketing for the first time using AdWords and various revenue share campaigns, and of course to do this sensibly we need to arrive at a reasonable estimate of LTV.

A: Makes sense!

Q: To calculate an adjusted LTV I removed all customers with a lifetime of less than 4 months but this gives a low estimate as this calculation ignores the bumper summer months and the extra paid for features put in place earlier this year.  Calculating LTV using ARPU and monthly churn (not sure how to calculate this in our environment) gives another different estimate.  Is there any help or advice you could perhaps give us?  If not in the US then perhaps you could recommend somebody abroad – can’t find anything in the literature relevant for start-up like us.

A:  It sounds to me like you’re trying to make this too complicated, at least for the place you are at this time.  Monthly churn and the “28 day” threshold are nice to know on a tactical level, but LTV is more of a Strategic idea that does not necessarily benefit from analysis at that level.  And you may not really want LTV, but a derivative that might be more helpful.

Continue reading Customer Value in the Freemium Model →