Category Archives: Web Analytics

Control Groups in Small Populations

The following is from the January 2010 Drilling Down Newsletter.  Got a question about Customer Measurement, Management, Valuation, Retention, Loyalty, Defection?  Just ask your question.  Also, feel free to leave a comment and I’ll reply.

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: Thank you for your recent article about Control Groups.  Our organization launched an online distance learning program this past August, and I’ve just completed some student behavior analysis for this past semester.

Using weekly RF-Scores based on Recently and Frequently they’ve logged in to courses within the previous three weeks, I’m able to assess their “Risk Level”– how likely they are to stop using the program.  We had a percentage who discontinued the program, but in retrospect, their login behavior and changes in their login behavior gave strong indication they were having trouble before they completely stopped using it.

A: Fantastic!  I have spoken with numerous online educators about this application of Recency – Frequency modeling, as well online research subscriptions, a similar behavioral model.  All reported great results predicting student / subscriber defection rates.

Q: I’m preparing to propose a program for the upcoming semester where we contact students by email and / or phone when their login behavior gives indication that they’re having trouble.  My hope is that by proactively contacting these students, we can resolve issues or provide assistance before things escalate to the point they defect completely.

A: Absolutely, the yield (% students / revenue retained) on a project like this should be excellent.  Plus, you will end up learning a lot about “why”, which will lead to better executions of the “potential dropout” program the more you test it.

Continue reading Control Groups in Small Populations

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss

Acting on Buyer Engagement

Over the years I’ve argued that there is a single, easy to track metric for buyer engagement – Recency.  Though you can develop really complex models for purchase likelihood, just knowing “weeks since last purchase” gets you a long way to understanding how to optimize Marketing and Service programs for profit.

Which brings me to the latest Marketing Science article I have reviewed for the Web Analytics Association, Dynamic Customer Management and the Value of One-to-One Marketing, where the researchers find “customized promotions yield large increases in revenue and profits relative to uniform promotion policies”.  And what variable is most effective when customizing promotions?

The researchers took 56 weeks of purchase behavior from an online store, and used the first 50 weeks to construct a predictive model of purchase behavior.   Inputs to the model included Price, presence of Banner Ads, 3 types of promotions, order sizes, number of orders, merchandise category, demographics, and weeks since last purchase (Recency).

The last 6 weeks of data were used to test the predictive power of the model, and the answer to which variable is most predictive of purchase is displayed in the chart below, click to enlarge:

Weeks since last purchase dominated the predictive power of the model, controlling not only the Natural purchase rate (labeled Baseline in chart above, people who received no promotions) but the response to all three different types of promotion.

Continue reading Acting on Buyer Engagement

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss

Choosing the Size of Control Groups

The following is from the December 2009 Drilling Down Newsletter.  Got a question about Customer Measurement, Management, Valuation, Retention, Loyalty, Defection?  Just ask your question.  Also, feel free to leave a comment and I’ll reply.

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

 Q:  I am a big fan of your web site and read your Drilling Down book. Great work!

A:  Thanks for the kind words!

Q:  I was wondering if you could help me picking the right control group size for a project of ours?  The population is 25 million telco customers that for which we want to do a long term impact analysis (month by month) in regards to revenue increase versus control group.  The marketing initiatives are mix of retention, lifecycle and tactical/seasonal activities.  We want to measure revenue increase through any of the marketing activities compared to control group.

A:   Great project, this is the kind of idea that can really improve margins if you can find out which specific tactics drop the most profit to the bottom line.

Q:   I have searched the web for some help and found calculators that say: On 25 million and smallest expected uplift of 0.1% and highest likely rate of > 5% the calculator gives 250k (1%).  Is that sufficient to calculate the net impact on the remaining base?  Would be very grateful if you could give me your thoughts.

A:  Well, it could be and might not be…

Continue reading Choosing the Size of Control Groups

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss

Customer Value in the Freemium Model

The following is from the November 2009 Drilling Down Newsletter.  Got a question about Customer Measurement, Management, Valuation, Retention, Loyalty, Defection?  Just ask your question.  Also, feel free to leave a comment and I’ll reply.

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: You kindly clarified a few issues when I was reading Drilling Down earlier this year – so I hope you don’t mind the direct email.

A: Yes, I remember!

I am working for www.XYZ.com, a social networking / virtual world site based abroad but visitors are 85% US.

Our growth up to now has been mainly viral and in the summer we hit 1.2M UVs operating on the Freemium model with only 5% of our registered users converting to paying customers and a significant portion of our revenue coming from ads.  On average our customers are active on the site for something like 4 months making their first purchase around day 28. 

But to take us to the next stage we are embarking on some marketing for the first time using AdWords and various revenue share campaigns, and of course to do this sensibly we need to arrive at a reasonable estimate of LTV.

A: Makes sense!

Q: To calculate an adjusted LTV I removed all customers with a lifetime of less than 4 months but this gives a low estimate as this calculation ignores the bumper summer months and the extra paid for features put in place earlier this year.  Calculating LTV using ARPU and monthly churn (not sure how to calculate this in our environment) gives another different estimate.  Is there any help or advice you could perhaps give us?  If not in the US then perhaps you could recommend somebody abroad – can’t find anything in the literature relevant for start-up like us.

A:  It sounds to me like you’re trying to make this too complicated, at least for the place you are at this time.  Monthly churn and the “28 day” threshold are nice to know on a tactical level, but LTV is more of a Strategic idea that does not necessarily benefit from analysis at that level.  And you may not really want LTV, but a derivative that might be more helpful.

Continue reading Customer Value in the Freemium Model

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss

“X Month” Value

The basic concept of LifeTime Value (LTV) was ably outlined by Seth Godin in a great post here.  If you know the average net value of a customer is $2500 over their “Life”, why would you not spend  $50 (or $200, really) to acquire each one?  As long as you stuck to the model, your company would be insanely profitable over time.

Their are 2 primary challenges to implementing this idea.

1.  “Over time” is a concept many management folks have a hard time embracing; what matters are the profits this year, or this quarter, or this month.  Unless the whole company embraces an “over time” measurement approach it is difficult for Marketers and Analysts to drive towards programs and practices supporting the LTV outcome.

2.  The $2500 is an average figure.  Most customers are worth less; 10% or 20% are worth much more.

Most people I talk to embrace the general idea of LTV models intuitively.  It’s really a cash flow concept, isn’t it?

So Financial people get it right away, and if Marketers could align with it, there would be no conflicts and the Marketing budget becomes virtually unlimited.

In fact, many folks in the PPC world follow just this model – they have unlimited budget as long as each conversion costs no more than “X”.  Because the company knows if it spends no more than X on a conversion, it always makes money.   Marketers and Analysts involved with these “Cost < X” PPC programs love them, because Management loves them. 

And Management loves them, why?  Because the CFO loves these programs  Why?  Because they are based on Cash Flow analysis, which CFO’s understand very, very well.

So then, what will it take to get more acquisition budgets like these Cost < X  PPC programs?  We have to address the two challenges above:

Continue reading “X Month” Value

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss

Awareness versus Persuasion

In the early days of Home Shopping Network (live TV, not online), we were doing some ethnographic research and started to find “physical clusters” of customers – neighbors or people who worked together.  For example, one of these groups was nurses at hospitals,  especially nurses  who worked the night shift.

We looked for the most active member of the cluster (our “thought leader”) and asked them if they would help us with a “member get a member” program.  Would they be willing to distribute discount coupons to their friends, especially ones who were not already customers?  Time after time, the answer was:

“Honey, all my friends are already customers of yours”.

We launched the program anyway, because it was a pet project from upstairs  – I was a junior marketer at that point so I couldn’t kill it ;)  The program never, ever worked, no matter how hard we tried.  It generated very few new customers while giving lots of discounts to people who were already active buyers.  Basically,  the cost of those discounts overwhelmed the value of the new customers generated.

Apparently a similar thing happens online with Social marketing.

As part of a WAA program that reviews academic research for WAA members, I was able to take a look at a paper titled:  Firm-Created Word-of-Mouth Communication: Evidence from a Field Test by David Godes and Dina Mayzlin.

Continue reading Awareness versus Persuasion

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss

Adoption and Abandonment

Out of the Wharton School we have a nice piece of behavioral research on the effect speed of Adoption has on longer-term commitment.  The article, The Long-term Downside of Overnight Success, describes research finding “the adoption velocity has a negative effect on the cumulative number of adopters”. 

This research dovetails nicely with a lot of the topics discussed here on the blog lately, so I thought I’d use it (with a nod to Godin’s post on Strategy vs. Tactics today) to provide some fodder for thought.

First, the importance of Psychology in Marketing.  So many of the “discoveries” arrived at through  brute force testing of Online Advertising are already well known in the greater discipline of Marketing through Psychology.  For more on this read “The Other 3P’s” and if you’d like to do something about lack of knowledge in this area, make sure to read this comment on source books.

Second, this research is a great example of isolating the true drivers of behavior.  The idea of looking at baby names to isolate the real behavior from “technology and other commercial effects” while including “symbolic meaning about identity” results in a broad, Strategic-level answer to the question, not a Tactical one. 

Why is this important?  It means the results can be applied across a host of different Marketing situations, rather than only a specific one. 

Continue reading Adoption and Abandonment

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss

Norms of Reciprocity

Social Marketing Doesn’t Rely on Social Media

Do you believe human beings share certain fundamental traits that define “being human”?

If so, do you believe that human beings tend to behave in certain ways under certain circumstances?

If so, do you then believe since human behavior has these tendencies, it can often be predicted?

If so, then do you think perhaps the study of Psychology and Sociology might provide you some clues to creating successful businesses, campaigns, products, and services?  While your friends and competitors are all iterating their way into oblivion?

On the web, time and time again, we see the same themes repeating.  Yet with each introduction of a new technology, these themes tend to be treated like a new discovery, even though the theme has been well established in the past.

Norms of Reciprocity is a constant human theme.  You may know the expression of these norms as “Sharing”.  Web old timers will probably recognize this idea as “Give, then Take” from the I-Sales discussion list as early as 1995.  In various forms, this theme goes back to the beginning of human history, all the way back to the handshake and other greeting gestures.  This same theme is embedded in countless Religions all over the world: “Do onto others as you would wish them do onto you”.  At least a couple centuries old, this idea.

Norms of Reciprocity simply means this: When you do something nice for a human being, help them in some way, this human tends to feel Gratitude towards “the doer” and tends to do something nice back.  Gratitude drives the desire to Reciprocate, because it’s just what humans do, it’s normal, a “norm”.

Norms of Reciprocity.

Continue reading Norms of Reciprocity

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss

Analyze, Not Justify

Does this issue affect the Web Analytics Maturity Model?

A conference call with a Potential Client last week jogged my memory on a couple of events that happened during the flurry of Web Analytics conferences this Spring.  Here’s a portion of the call…

PC: “We’ve tried proving the profitability of our Marketing efforts and can’t seem to get the numbers working correctly.  So Jim, what we’d like you to do is take all this data we have, and justify the Marketing decisions we’ve made by proving out the ROI.”

Jim: “I’m sorry, did you say justify?  To me, justify means “find a way to prove it works”.  Is that what you are asking me to do?  Wouldn’t it be more beneficial to analyze the results, and then optimize your Marketing based on these results?”

PC: “Jim, around here we’re pretty clear our Marketing works, and Management knows this.  But Finance is asking for some backup, some numbers to justify the spend, not to analyze it.  We don’t need analysis, we need your ‘expert credibility’ to help us out with this.”

Jim: “I see,” thinking this is not a job I’m going to enjoy.  It’s the old ‘buy an outside expert’ routine, which I detest.

PC: “Jim, the team is united behind this mission, are you on board?”

Jim: “Well, perhaps I could be on board, as long as what you want is an analysis, which may also justify the decisions you have made.  But it might not, so I just want to be clear on what…”

PC: “You  know what Jim?  I don’t feel we’re going to have a fit here, I’m getting you’re not a team player.  Thanks for your time”.  CLICK

Sigh.  I’m actually grateful they hung up, I really dislike explaining to people why I won’t work with them.

Continue reading Analyze, Not Justify

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss

Hacking the RFM Model

The following is from the May 2009 Drilling Down Newsletter.  Got a question about Customer Measurement, Management, Valuation, Retention, Loyalty, Defection?  Just ask your question.  Also, feel free to leave a comment. 

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q:  First of all thank you for your help.  I have some questions I would be pleased if you answer them for me.

A:  No problem!

Q:  1. RFM analysis – is it possible to use some other ranking technique rather than quintiles? Using quintiles for bigger databases will cause many tied values, isn’t it a problem?

A:  Sure, you can use it any way it works best for you.  There is no “magic” behind quintiles, you can use deciles or whatever works best. It’s the idea of ranking by Recency, Frequency, and Value that is the key concept in the model.

I’ve seen dozens and perhaps hundreds of variations on the core RFM model, depending on how you classify a “variation”.  One change that’s common is changing the scaling, as you mention above, to accommodate the size of the database.  Smaller databases use quartiles or even tertiles.  Larger databases, choose the ordered distribution that meets the need.

Continue reading Hacking the RFM Model

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss