Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: I purchased your book and have a few questions you can hopefully help me out with.

A: Thanks for that, and sure!

Q: We have 4 product lines and 2 of them are seasonal. i.e we have customers that year in year out purchase these items consistently but seasonally, for example, every spring and summer.  Then they are dormant for Fall and Winter.  Should I include these customers along with everyone else when doing an RFM segmentation?

A: Well, it kind of depends what you will using the RF(M) model for, what kinds of marketing programs will be activated by using the scores. If you know you have seasonal customers and their habit is to buy each year, AND you wish to aim retention or reactivation programs at them, I would be tempted to divide the customer base so that seasonal customers are their own segment.  Then run two RF(M)  models – one for the seasonals, and one for everyone else.

Q: If I include seasonal customers, and I run RFM say on a monthly basis, these seasonal customers will climb / fall drastically with time depending on the season, so it seems like it may complicate the scoring process.

A: Sure, and you could segment as I said above.  Or, you could run across a longer time frame, say across 2 – 3 years worth of data. This would “normalize” the two segments into one and take account of the seasonality in the scoring – perhaps be more representative of the business model.  However, the scores would become less sensitive due to the long time frame so the actions of customers less accurately predicted by the model.

Q: Can you provide me with some examples as to how segmentation is carried out?  Let’s say I being with RFM and all my customers are rated 5-5, 5-4, 4-5 etc.  What are the next steps, do we overlay with other characteristics like age, gender, etc?  Or are the 5-3 etc. our actual segments?

A: This goes back to what you want to use the RF(M) model for.  In the standard usage, each score will have roughly the same number of customers in it, those with higher scores will be more likely to respond to marketing and purchase, lower scores less likely.

Another way to say this is: since the lower scores are less responsive, they require higher value discounts or promotions to activate them.  So you can customize offers based on score, which will provide the maximum response for the minimum discount exposure.

Try testing different discount levels with different scores to see where you get maximum profit; an example of that kind of testing and math is here:

Using Recency to Drive Promotional Profit

The above assumes you are focused on response and profit, two of the more common objectives in data-based marketing ;)

However, other valuable information can be discovered using this framework, see chapter 20 about adding customer characteristics to RF(M) scores.  This is not needed if you are just concerned about response / profit for campaigns – the segments are the segments.

But if you’d like to know, for example, what kind of merchandise is appealing to 5-5 (ultra-best, highest responding) customers, you could run the scoring, take the 5-5’s, and then cross-tab with whatever else you have – find out what kind of merchandise they buy, what their sex is, etc. – any kind of data you have or can get on them.  This kind of work can help develop creative, decide what to feature on the home page, etc.

Q: Should we provide them a segment name like High Recent / Low Frequents, Low Frequents / Low Recents etc?  How do the other characteristics come into play with respect to the naming conventions, segmentation etc.  I keep reading of different segments like ‘Loyalists’, ‘Laggers’, are these defined through our subjectivity?

A: Generally yes, subjective labels; you may  find it is easier to communicate using labels rather than scores, so you could suggest that certain ranges of scores be called something like ‘Loyalists’ or ‘Laggers’.

The High Recent / Low Frequents segmentation is the quadrant approach from LifeCycle mapping, see the chapter 22 in the book.  This is an alternative way to use the RF(M) variables to map your customer base in a more Strategic way than RF(M) affords.

In other words, you could use RF(M) to score for Campaigns and the LifeCycle maps to report on your progress over time to management, since both approaches are based on the same variables – great way to “connect the dots” for manic-ment.  Also, RF(M) is a bit hard to visualize just using the scores; the LifeCycle maps allow you to plot campaign results in a very visual, easy to understand display of the data.

See these newsletters for a deeper explanation of Strategic versus Tactical use of RF scoring:

RFM versus LifeCycle Grids

LTV, RFM, LifeCycles – the Framework

Hope that helps!

Jim

===================

Q: How do I establish the optimal number of segments / buckets I should have when analyzing a customer database?  I was using a method of dividing them into deciles and assigning Decile 1 to Segment A, Decile 2 & 3 to Segment B, Deciles 4 – 6 to Segment c and Deciles 7 – 10 to Segment D.  However, the absolute number in each is not practical as I will not be able to manage that number of customers in the way I need to.

A: Well, it would help to know what Kind of business this is, but in general, the most effective place to break behavioral segments is where behavior changes.  You can force people into Deciles, which often does not create meaningful segments, or you can look at the data and see where there are “bulges” or changes that seem significant.

So, for example, look at this LifeCycle Grid:

Current / Potential Value Matrix

Note the low Current Value breaks of 1, 2, 3 because there are large groups of customers there.  Then, 4 – 9, because each of those has many fewer customers and their behavior is similar.  Same with 10 – 24, that’s a pretty good customer comparatively, and after 25 purchases, well, they’re crazy-good customers and behave similarly.

These cutoffs start out as somewhat arbitrary, but over time and testing you find out, for example, that a 9X buyer behaves more like a 5X buyer and not so much like a 10 – 24X buyer, so the cutoff is 9X.

Here’s another example:

Freemium Customer Conversion

The segments graphed in this post are actually deciles of 1 year customer value; but the data plot is of the first 14 weeks of weekly spending.  Note how there are clear inflection points where the  behavior changes, and this is where I would generally create the “boundaries” for a segment.  Doing it this way means the segmentation already has some type of powerful behavioral trait, making it by definition significant and “actionable”.

Hope that helps!

Jim

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Defining Behavioral Segments

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: We’ve been playing around with Recency / Frequency scoring in our customer email campaigns as described in your book.  To start, we’re targeting best customers who have stopped interacting with us.  I have just completed a piece of analysis that shows after one of these targeted emails:

1. Purchasers increased 22.9%
2. Transactions increased 69%
3. Revenue increased 71%

A: There you go!

Q: My concern is that what I am seeing is merely a seasonal effect – our revenue peaks in July and August.  So what I should have done is use a control group as you described in the book – which is what I am doing for the October Email.

A: Yep, that’s exactly what control groups are for – to strain out the noise of seasonality, other promotions, etc.  But don’t beat yourself up over it, nothing wrong with poking around and trying to figure out where the levers are first.

Q: Two questions:

1.  What statistical test do I use to demonstrate that the observed changes are not down to chance

2.  How big should my control group be – typically our cohort is 500-800 individuals

A: Good questions…

On a group that small, you are probably not going to get anything “statistically significant” without ruining your total profit, e.g. might have to use 50% in control.   If you have the leeway to do it, that’s what I would do.

On the other hand, in some cultures people will go bonkers over giving up sales to learn something really important.  OK, so take 10% as control and repeat it 3 times; if the results are stable then you have your proof.  Do another control every once and a while (every 6 months?) just to make sure it tracks.

Either way, you don’t really need statistics.

Practically, confidence is the likelihood a sample represents the population.  This can be a really useful idea when you are forced into very small sample sizes or the event is highly risky to repeat.  But here, if you are testing a really large slug of the population, confidence is less useful.  Or if you can repeat the event (because essentially, you are in control of it and it’s low risk), do you really need to force yourself through the screw of complying with the statistical math?  It’s like using a 727 for crop dusting, overkill for the situation, methinks.

If you were running a drug manufacturing line, statistical concepts like confidence and significance are absolutely valuable.  But for a marketing program?

That’s why I love the idea of “beefy controls” in start-up projects because I *do not* have to rely on statistics that the audience likely does not understand and provide room to question the results, e.g. “Yea, but what if the result is an outlier?”  Very appropriate in high risk situations, with giant populations and a lot of money on the line.  For this situation, perhaps not.  But, if you’d like to go that way, there’s lots of calculators on the web that let you play with some of the numbers anyway.

Here’s one, make sure to read the descriptions of the variables underneath the calculator:

http://www.surveysystem.com/sscalc.htm

Nice work on the core campaign idea, by the way!  Now we just have to tighten it up a bit…

(3 months later)

Q: We decided to tighten the targets and do a “best customer defection” email program. Basically, we look at customers who has an RFM score of 555 in the previous scoring period who have dropped out of that score.

A: Interesting!  So instead of targeting by  guessing the current score of a defecting best customer (say 355), you are looking for all customers who were formally best customers, regardless of current score.  This is a subtle difference, but much more of a LifeCycle approach and frankly why I prefer these kinds of ideas over “straight” RFM.

An example might be helpful.  Let’s say the acquisition folks run a huge new customer campaign in between the prior RFM scoring and the scoring done before your campaign drop.  A big inflow of new customers can artificially “force” certain groups of customers down in score – even though their own behavior has *not changed*.  In this case, the new score is not reflective of actual behavior, so increases noise in the system.

That’s the problem with the “Snapshot” or date-specific view of Customer State – it’s a single point without reference.  By using prior score, you are acknowledging behavior over time and the primary importance of the former State, as opposed to the current State – a Movie as opposed to a Snapshot.

In other words, from a  Marketing perspective, I’m more interested in the path they are taking through the LifeCycle than any particular point in time during the LifeCycle represented by a single RFM score.

Q: Good news on your advice.  We ran a 50% control (500 purchasers in each group) and the results really nailed the issue for us. The actual number of purchasers remained unchanged at 20% but Total Revenue and Average Spend increased by 40% compared to control.

(Jim’s Note: for those not following, a very precise target group of 1000 was split into 2 groups of 500.  One group received this  campaign, the other did not.  People who did not receive the campaign purchased at the same rate as people who did receive the campaign, but the people who received the campaign averaged 40% higher spend).

A: Awesome.  So what you are seeing is Customer State makes a huge difference in terms of what offers / timing can be most effective for this “Recently defecting best customers” cohort.  If I’m reading your numbers correctly, no lift in response versus control but a huge lift in revenue.

To me, that means these customers are early in the process of defection – still buying, but without a special treatment, slowing down the monthly spend.  After all, they are very Recent (former 5XX), so highly likely to purchase again, which is why lift in response was flat – they likely would have purchased anyway.

Not a bad time to hit them.  Offers to a very Recent State should focus on increasing order value, not generating response – you don’t want to spit into the wind, but go with the natural flow of the behavior.

In other words, these customers likely would have purchased anyway, but at lower price points if they had not received the campaign.  The common way this is addressed is with  “threshold” discounts – if average order is $50, then something like “$10 off any purchase over $50″ – test different thresholds to maximize profitability.

Looks like you gave them the right offer ;)

On the other hand, a straight discount to this specific best customer group – $10 off anything, and especially when their normal category of purchase is promoted to them – almost ensures that you will lose money.  Why?  Most of these customers would have bought at full price anyway, as demonstrated by equal buying activity whether the customer received the campaign or not.  So the discount turns into a loss versus no campaign at all.

Unfortunately, I see a lot of this exact type of campaign delivered to best customers because all customers get some version of the same offer.  “Hey Jim, we’re not sending the same message to every customer, we send different messages by segment”.  Sure, the copy and art are customized for different segments, but the segmentation is not by Customer State, so the offers are mismatched and suboptimal.

This is the value of using control groups; they drive understanding of Marketing concepts like opportunity costs and subsidy costs.  These two concepts are the reasons why ignoring Customer State is suboptimal: by not segmenting using State, you will get lower than possible profit or sales at most customers, depending on Customer State.

Had you not delivered a campaign tailored for prior Customer State, money would have been left on the table by way of lower order size.  And 40% Revenue lift sounds like it might have covered the cost of the campaign ;)

Q: We tried to run a Student T test on the results but our new statistician informed me that the distributions were not normal – so on her advice we ran a Wilcoxan Test which gave us a highly significantly p = 0.016

A: Oh, so you still went the stats route?  Well, the fact you HAVE a statistician tells me the culture there is more familiar with interpreting these ideas, so more power to you.

Glad it worked out and keep me informed on how things go downstream.

Jim

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Increase Profit Using Customer State

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: Thank you for creating this useful website!

A: You’re welcome!

Q: When figuring out retention rate for an annual or a 8 months life time cycle period, how do I pick the starting period?  Do I look at their first orders on a date?  Or I pick a time frame such as one month?

A: It depends on:

1. What kind of “retention” you are talking about, the definition, which is probably impacted by the audience for the data

2.  What you will do with the retention data, what kind of decisions will be made and actions be taken because of the data

You should always ask these questions above  when someone requests “retention data” – or any other kind of analysis, for that matter!  For example, there probably is a huge difference in what you would provide to the Board of Directors for an annual benchmark and what you would provide to Marketing people for executing campaigns.

In the first case, the data would probably be used to inform Strategic decision making, for example, should we change our product mix or approach to pricing given the market?  In the second case, the data would probably be used in a Tactical way, for example, to target new customers who are predicted to defect because of the campaign they responded to or the product they bought.

If providing data to the Board, “annual retention rate” would probably make the most sense (again, you should ask, what’s it for?). If that’s what they want, you would pick a starting period, probably aligned with the fiscal year (Jan – Dec?), and find out what percent of people who purchased Jan – Dec 2009 also purchased Jan – Dec 2010.

That’s the annual retention rate.  Useful information, perhaps leading to the Board requesting action of some kind.  But by itself, you really can’t “do” anything with this data, there’s no source or targeting information, there’s no customer value information.

However, if you segment by campaigns, product of initial purchase, price points, offers, or other actionable variables, the retention rate could be just about any formula, e.g. what is the retention rate:

a. Today, of people who made their first purchase in 2005?
b. End of 2009, of people who made their first purchase in 2005?
c. Today, of people who ever bought Product X as their first purchase?
d. Today, of people who bought Product X as their first purchase in 2009?
e. Today, of people who had at least 2 service calls in 2010, who became new customers in 2009, who used a 50% off promotion?

and so on.  Retention rate for anything tactical almost always requires and audience and time frame to be defined.

Q: You mention in your article, “Total number of customers” as the denominator for calculating the customer retention rate, do you mean the total customers at the end of the period?  Or those total customers came in on the first date of a fixed period?  Or the first fixed period that I’m observing?

A: Whatever definition is the correct definition depending on the need of the audience.  There is no standard, other than perhaps the very first one, the Strategic “reporting” idea of year over year retention.  This is commonly used in reporting to Wall Street, for example.

While discussing this particular idea of “customers”, one might encounter the common problem of not knowing the definition of a customer, at least in terms of retention.

When does the company declare a customer is no longer a customer?  Is a customer  “everyone” who has ever purchased?  If the company has been around 10 years, and you are calculating retention rate “today”, as in how many of these total customers purchased in the last year, you may find you have a very low number, one that won’t mean much to anybody, and is not actionable.

On the other hand, if your definition of “customer” includes a level of activity, for example, “must purchase at least twice, one of those purchases in the past 3 years”, now you are talking about a highly actionable kind of retention definition.  Why?

Because there is some hope that people who have purchased at least 2x (Frequency), at least once in the past 3 years (Recency) could actually still be customers, as opposed to defected customers.  If I am calculating a “serious” retention rate, something to be used to take Marketing action, or pay out bonuses, or revise policies, I want to measure against people who actually have some Potential Value, some Value to the company in the future.  That’s how I define a customer.  To me, there isn’t any point in calling someone a customer who is unlikely to contribute to profits in the future.

If you define as a customer “anyone who purchased over the past 10 years”, you just have a dead metric that really does not reflect the reality of what taking action might produce.  In other words, you are including people who are extremely unlikely to still be customers, so what’s the point of the “customer retention metric” you created?

Does the above help answer your question?

Q: I wasn’t expecting you to reply me so fast and in such detail!!!  Thank you so much!  I’m calculating this retention rate for marketing and your answer is very helpful for me!!!

A: Great!  So maybe ask them specifically how they want to look at it, and if they seem puzzled, suggest to them various options.

I can tell you from experience with businesses like yours is the buying behavior tends to peak early and you have to act quickly if you want to extend the lifecycle.  Perhaps not quite as time-critical given your “triple bottom line”, but probably not too different.

This argues for a tighter leash on the definition of a customer, perhaps purchased at least twice, one of those past 6 months.  You could also do 2x purchase, at least once in past 3 years, and compare, it will give them a feel for customer defection trend / rate.

The next step would be the Lifecycle map, which uses Recency and Frequency in a more actionable way, like this example.

Marketing people should be able to use this map to target specific groups of customers, e.g. purchased 4 – 9 times, but not in the past 90 days.  These are good customers who are in the process of defecting, and require special attention to keep them on board.

After all, the point of measuring retention is not retention rate itself, it’s about increasing the productivity and profitability of the business system.  Just as you can optimize for conversion, you can optimize for retention, and sometimes you discover they conflict.

For example, one company I worked with featured certain products on their home page because those products had a high conversion rate on visits to the home page; they had “optimized” the home page for this scenario.

However, a very quick and simple calculation showed these products generated customers  with terrible repeat purchase rates relative to just about every other product with volume.   A quick survey of these customers found out why the repeat purchase rates were so low – almost all customers disliked the product and thought the company deceived them.  Turns out the company “over-sold” the product – and that’s why the high conversion rates.

In another case, PPC campaigns had been optimized for conversion without regard to customer retention.  Under a budget crunch, the lowest converting campaigns were killed, but overall sales volume over the next 3 months dropped much more than the sales generated by these campaigns.

Reason?  These low converting campaigns generated the company’s very best customers in terms of 30-day, 90-day, 180-day value, while most of the highest converting campaigns generated low value, single purchase customers on the same time frames.

This kind of analysis is simply not that difficult to set up and execute, relative to the extreme amounts of value that can be created:

1.  Pass campaign codes / info with the order to the backend order processing.  If you are not doing this yet, start right now!

2.  Select a campaign, choose a time frame.  If you want to match up to financial statements (a good idea if you will be talking to C-Level folks), say January 2010.

3.  Grab all new customers who came in on Campaign X during Month Y – what is their average value 1, 3, 6, 12 months later?  This is a Lifecycle by Campaign analysis, similar to the LifeCycle map example mentioned above.

The new customer experience (channel, offer, product) is one of the most powerful predictors of future customer value, and the value of these new customers relative to each other tends to remain stable regardless of how many other generic campaigns (weekly email) you throw at the customer over time.

Across all campaigns, about 60 – 80% of these new customers will have the same value at 12 months they had at 1 month.  The question to answer, as with any optimization, is this: knowing the customer value created by these campaigns varies widely, are we allocating the acquisition spend optimally?  For example, are we spending 70% of the budget to generate  20% of the annual customer value?  Are we willing to pay more for clicks that generate new customers with 10X higher annual value?

Retention rate isn’t just some mystical number, retention rate quickly turns into profit dollars and can have incredible financial impact!

Jim

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Optimizing for Customer Value

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: First off, I very much appreciate you sharing all this wonderful content on your blog and conferences such as eMetrics.

A: Thanks for that!

Q: My question is a simple one, but I think the answer may be hard: When does a visitor “need” a coupon?  *Need* defined as: visitor would not have placed an order unless presented with the coupon.

A: Hmmm…methinks we’re going to have to define a few concepts and be clear on the goals to make sure we are nailing this down… visitor versus customer, sales versus profit, etc.  In other words, answer is not hard, but could be complex without defining context.

Q: It’s still a mystery to me why so many retailers seem more than willing to hand over all their margins to Groupon or give coupons to basically all visitors.  I am curious whether you would approach this question using  observational data (eg web analytics) or experiments (eg AB testing), or both.

A: Right – is a mystery to me too!

There are certain situations where this approach might be appropriate, but the problem with much web “marketing” (which often is really just advertising without much thought about marketing) is often there is success in a narrow or special situation.  Then the pundits jump on and say “if you’re not doing this you are stupid”, regardless of the business situation and / or without recognizing the special circumstances that are driving success.  This is all the real Marketing stuff people leave out; understanding why it works, under what circumstances, for which segments, involving which products.

When you don’t really understand what is happening and why, no learning takes place.  When no learning takes place, everything is a “new idea” and people are surprised when the outcome is different.

So, for example, if I was launching a brand new service business (restaurant) or a new product that is complex and won’t sell without trial (like yogurt that “naturally regulates your digestive system”), Groupon might be a slam dunk for a product launch.  No surprise here; coupons are often used to drive trial in product launches because the need is to reduce price resistance and drive sampling.

On the other hand, if my product or company is well known and I have tons of loyal customers, Groupon could generally be a financial disaster if you care about profits.  But if all you care about is response to the coupon, it could be a great success!  Because tons of people who would have bought at full price anyway get a huge discount and you get to sell the product below cost.  Awesome!

What do I mean by this, how can you have high response and low or negative profits?

Here is what I have seen over the years: whenever response rates are abnormally high, it means you have a high percentage of responders who would have bought anyway without the coupon.  This is seen over and over in database marketing, online and offline.  From a financial perspective, it means you have probably given up the coupon value with no benefit, a so called “subsidy cost”.

How do you prove this is happening in a promotion?  If you want to really look into and prove these effects, first examine the percentage of response that is from current customers.  If it’s high, that’s the first clue the discount is cannibalistic, not incremental.

If you want to quantify this subsidy cost a bit more is a relatively simple way, take the customer redeemers as a group and look at their average sales for a few months prior to the coupon promotion, during the promotion, and a few months after the promotion.  Often what  you will see is their spending behavior changed very little during the promotion.

For example, let’s say the coupon is 50% off.  The monthly net spending sequence over time might look similar to this:

2 months prior: $100
1 month prior: $100
Promotion month: $50
1 month after: $100
2 months after: $100

This shows the customers redeeming the 50% off coupon did not change their behavior at all; they simply took the discount and bought what they would buy anyway.  Meaning, the coupon cost is a real cost to the bottom line with no offsetting incremental profit.

Bottom line, for every response you lost $50 in sales plus the cost of the campaign, even though you had tons of responses and sales from those responses.  If it was a large campaign, your overall sales for the month net of discounts probably dropped.  Financially, if your cost of goods is 50%, you gave up $25 in profit for every response, minus the per response cost of the campaign.

And, the above behavior is most likely to occur with best, most active customers!  Across all redeemers, you might get $60 or so instead of $50 during the promotion month, but you are still losing money on every redemption – the higher the response, the more money is lost!

This is one big difference between Advertising and Marketing.  Marketing goes beyond Advertising, wants to understand the relationship of specific products to segments of customers, how pricing and modes of distribution affect this relationship, and the profitability of the relationship.

So, with that backdrop, let’s try the question:

Q: When does a visitor “need” a coupon?

A: If I take your question literally, there is a concept in Marketing called coupon proneness, and it’s the classic definition of “needs a coupon”.  Essentially, it means the more coupons you give people the less likely they are to buy without one. If you can imagine what this looks like over time, it’s margin erosion hell.  It’s taking the example above, where no incremental profits were generated, and ensuring it will happen time and time again.

From a Brand perspective, always offering coupons means you are teaching people your prices are too high, or there is a tangible reason your company has to “beg” for sales (implies poor service or quality).  Either way, the outcome is not so good for Brand trust and any evangelism that might result.

The exception to the above is among the “never pay full price” segment, who don’t buy anything without a discount / coupon.  From this segment, you get the benefit (?) of your coupon offers being spread all over the web, attracting many other “never pay full price” customers who generally have negative net values to the company.  Great, huh?

The end result of this pattern is horrible customer loyalty, margin erosion among current customers, and lots of new customers that are 1x buyers.  This means you have to spend *even more* on Advertising to constantly chase new 1x customers, while at the same time your margins in the current customer base are being consistently eroded.

Certainly not an optimized system!

Some people will argue the “extra sales” they get are worth the price of encouraging the above behavior.  But there are not too many businesses that put sales in the bank, what they put in the bank are profits.  So this is very short-term thinking and in fact, you find a lot of businesses that follow this model perform very poorly financially.

So, you ask, when would a visitor “not have placed an order unless presented with the coupon”?  The answer is this: when you have “presented” a coupon before, and the more often you have done this, the less likely they are to buy *until* you present one.

Sure, you could use A/B testing, but it’s not hard to guess what you will find – when you present a coupon, more people buy.  Duh, that’s Advertising, right?  But that’s optimizing for conversion, not for profits, and conversion can’t be deposited in the bank any more than Sales can be.  You have to go further.

For example, if you had the capability to recognize purchase or visit patterns among visitors, you could segment by these behaviors and present coupons on the site only when they were likely to have an incremental rather than cannibalistic outcome.  For example:

1. A new visitor who becomes a repeat visitor X times but does not buy

2. A current customer who has not purchased in over X weeks

and so forth.  You could test for “X” and optimize for highest profitability if you also ran a “null” control group  – where if A = coupon, B = no coupon.  Then look for incremental sales  behavior or “lift” from those offered the coupon versus those not offered a coupon, and run out the profit and loss.

Of course there are other scenarios, mainly current customer comes to your site because  you sent them a coupon, as opposed to presented one on the site.  Not sure if you were including that in your question, but I took your meaning literally.

The scenario with subsidy costs when sending customers a coupon is basically the same as the example above, except you control which customers get what coupon values or if they get a coupon at all.  More info on executing and measuring in that scenario for customers here, and for an example of a company putting this approach into practice, see here.  You can really drive higher profits by doing this correctly.

Jim

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

When Does a Visitor Need a Coupon?

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: I was wondering if you’ve done any work with, or given thought to, companies who have a cloud based Freemium business model?

Should they be tracking usage (or anything) at the free level?  Should they be tracking usage at the paid level?  I’m sure defection rates are a big problem, but I’m wondering how many focus on engagement thru mass marketing versus trying to keep what they’ve got, or influence the free users to make the leap to paid.  Any thoughts on this?  Maybe you could do a blog post on it.  It seems like a good fit with your brand of analysis but I’m just starting to think it through…

A: I just finished an analysis that’s a good example of this problem.  Behavior during the Freemium period can predict who is highly likely to become a paying customer, who will need marketing efforts like additional sampling / package discounts, and who will not become a customer no matter what you do.

So the answer is you need both, analysis of paid and free.  But in particular, what you need to do is understand the transition from free to paid by comparing the behavior of known converters versus non-converters over time, preferably using events that create value for customers.

Typically the differences will be volume / persistence related, generically, low Recency high Frequency.  Also, likely converters to paid will tend to use a wider variety of features consistently.  In the analysis, the question to answer is which of these value-creating behaviors is predictive of becoming paid?

Said another way, you tend to see a fairly fast drop-off among *all* new Freemium customers after the initial burst of activity, but the ones that are not going to stick tend to drop off even faster in their activity.  Then, there is a “bounce” in activity where the ones who are most likely to end up as paid begin to cycle behavior more quickly and begin to use more features, and the others simply drift off the map, with no “bounce” as Recency becomes extended.

Classic LifeCycle analysis – customers tell you what they will become in the future by what they do today.  Having the very detailed behavioral information typically seen with interactivity just multiplies the ability to do this kind of prediction.  More on the Freemium model, including determining appropriate cost to acquire, is here.

Q: Do the standard analytics packages allow a business to look back at the “free” behavior of paid subscribers?  I’m thinking of Freemium cloud based solutions and how they would track this.  Do products like Crazy Egg get you there or do you really need something more sophisticated to do this kind of analysis?

A: I’ve never used Crazy Egg so I don’t know about that one specifically, but in general you can do quite a lot with the basic tools that support customizable segmentation.  The challenge with going that way is you have to be super-technical with the implementation to capture important event data points, you have to create many different segments, and then the killer problem – you can’t “re-analyze” a different approach with these tools, for the most part.  If that’s what you mean by “look back”, it’s highly unlikely you could accomplish what you need to do.

So it’s possible, but these tools are not really designed for “behavior over time” work and certainly don’t allow for much “exploration” of the data – any change in analytical approach is likely to be a “going forward” type of measurement, not looking back.  So there would be lots of iteration before you even knew if the analytics set-up was correct or what events are meaningful.  In other words, it’s possible but could waste a LOT of time.

I’d much rather find the system that contains the key elements of activity – when did they sign up, what features are they signed up for, when did they add other features.  This data probably resides in whatever system manages the account.  Dump that data off into a spreadsheet or database, try to figure out what’s meaningful, look for correlations.

Then, once you have a grip on some solid ideas, then maybe you go into the front end and try to align traffic and behavior with known “events” that seem to predict upgrade to pay, if that’s what the mission is.

Otherwise, you will be setting up a ton of tracking on all kinds of events not knowing what is meaningful, and then dealing with a really poor interface for the analysis of those events.

The other way to go, of course, is to use one of the advanced web analytics tools, which sit on real databases and can be queried.  But assuming that’s not an option, I would try to look for hard data points in the backend first, then knowing key behaviors, look for what might cause those behaviors in the traffic side.

Example

Let’s say you have a project management application that has a 60-day free trial then converts to paid.  Value is created for the customer when they use the functionality of the app – say create project, comment on project, upload file, or any other actions you deem critical.  “Traffic” in a situation like this may be only marginally indicative of value creation; rising activity could even be a negative indicator (frustration with application).

So, you want to create a situation where you can analyze these important behavioral events by account, and (ideally) you want to know the source of the account creation – campaign code, referrer, etc. That’s all you need for data, simple table, maybe a dozen columns.

Then, compare average account that converts to paid with average account opened at the same time as the converters but does not convert, over the 60-days before trial end.  For each of converting and non-converting, aggregate each of key events by week, divide by number of accounts to get average behavior per account, and you would have 8 weekly average data points for each of the events, both for non-converting and the converting accounts.  Maybe a dozen simple line graphs with 8 weekly data points, one set for accounts that paid, one set for accounts that did not.

Converting and non-converting graphs should look different for some variables.  Both will typically start out with high levels of activity, then for some variables you will see them diverge.  This not only predicts which variables affect conversion, but reveals to you the best time during the 60-days to intervene with surveys, help, or other marketing programs to re-engage the accounts that appear to be headed for defection.  If you have campaign data, also which campaigns tend to create accounts that convert and which don’t.

One of the event graphs may look to be more predictive than the others, with abrupt changes in direction going into the conversion event.  For example, perhaps it will look like this:

(Click pic for larger image)

This is the behavior of 10 different 1st year spend levels (deciles) over the first 14 weeks of their Life, engaging in an event that creates value for them.  The dark blue line represents average top spender.  Note how for top spenders, the profile is quite different.  The graph tells you that by week 4 or so, you can probably predict who will become a best customer and who will need intervention based on this activity.

You can run this kind of event profile for any variable – events, campaigns, etc. as long as you know complete / non-complete goal or end value of the customer.  In your case, since the goal outcome is binary, there would be 2 lines instead of the 10 spending deciles above: converters versus non-converters.  Create a converter versus non-converter chart for each key activity variable (create project, comment on project, upload file, or any other actions you deem critical), and look for this kind of divergence.

Drilling down more deeply by excluding all but 3 lines so we can see the behavior “in the middle”, we find some interesting patterns:

(Click pic for larger image)

Here, we see patterns that provide clues to the testing targets one might want to address to see if “middle” customers could be turned into better customers.  The blue segment, showing a series of higher highs and higher lows after it “bottoms” for this behavior, is most likely to benefit from intervention of some kind.  The pink segment looked promising, but then put in lower highs and lower lows – these customers lose momentum quickly and have trouble self-sustaining.  The yellow segment was never really in the game at all.

Yes, the comparison to stock market charting is intentional!  It’s an expression of group behavior.

If I had to pick the segment with the best potential, I would try the blue segment first, and the data points could be used for automated triggering of different types of campaigns. For example, “If by week 4 activity for Variable X  falls below 60, trigger Campaign A.  Then if by week 11 activity for Variable X is not above 40, trigger Campaign B.”  Remember, these are averages, so not all customers in the segment are below threshold.  The idea is to target a specific behavior with a specific message.

Just to be clear, you don’t need the goal value of the customer to put a model into practice, only to prove the initial model – certain patterns in behavior predict high value customers.  Once you know the end value of  customers – convert or not, monetary value, any goal – you can run the LifeCycle movie “backwards” like the charts above and find out which early  behaviors are predictive of high and low value customers.

If you want to go further, you could show these graphs and data to a modeler and see if they can create a more precise mathematical model, which can be developed much more quickly with this kind of evidence to review.

Once you fully understand what this LifeCycle landscape looks like, THEN you could go back and instrument the web site and analytical tool to monitor some version of this data in a more automated way.  But trying to guess what’s going to be important beforehand and work through a study like this using a vanilla web analytics tool is the very, very long way to get where you need to go!

Jim

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Freemium Customer Conversion

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: One of the first things I am doing in my new job is to identify the Customer Lifecycle pattern – how many periods (month, year) will it be before a customer is likely buy again.  In enterprise software industry, where software cost easily 6 figures, # of years is a reasonable time frame.

A: Yes, one would assume this.  But these notions would most likely be based on a feeling of the “average” behavior, and on average, it probably does take a long time.

What is not known is this:  if the “average” is composed of short-cycle and long-cycle buyers, who are the short cycle buyers, and what are they like?  What industry SIC code, for example?  And can we get more of them, or at least focus more resources on them, if they are the most profitable?  So the challenge is not only to look for the “average”, but then understand how this average is composed.  If you can break down the average by industry, or by salesperson, for example, this might be highly directional information.

Q: From my internal analysis, however, I discerned from the sales figures something quite counterintuitive – the period between first and next sale is much shorter than I would have thought for the SW industry in general.

A: Pleasant surprise, eh?  I don’t know what kind of figures you are looking at, but make sure the data is in fact what you think it is.  For example, if you want to study software purchase itself, do the “sales” figures you are looking at include transactions involving not the sale of software, but also service, like installation or modification fees?  It would make sense that a “software sale” would be followed pretty quickly by an “installation” sale, for example.  You need to know this to properly segment.

Q: Would you be able to point me to some studies on how often customers wait after the first purchase before contemplating an upgrade of software or something you personally have done in consulting projects for SW companies?  This industry benchmark will then shed some light on whether this trend is something peculiar to our company or not…

A: I am not aware of any published study of this type.  And as you might imagine, these numbers would vary quite widely in the industry and the nature of the information would be a highly guarded corporate secret.  So I don’t think you will find any “benchmark” studies of this type, and sorry, I can’t share client data with you!

Q: The next step for me then is to map out the drivers for this behaviour and then calculate the LTV (LifeTime Value) and take a look at the actual LifeCycle events creating this LTV.

A: Yes, but to be precise, LTV is typically a forecast when working with current customers; it’s not known until the customer actually defects, marking end of  “Life”.  What you are probably looking for is more accurately called “Life to Date” (LTD), the actual sales of a customer from start of relationship until present.

Also, when segmenting customers by LTD, of course look for temporal bias – a 10-year customer is likely to have higher LTD than a 2-year customer.  If there are enough customers, it might be a good idea to first segment by start year, then LTD.  This way you have cohorts of customers who are going through the same experiences together and differences in LTD will be more significant in terms of predictive power – you don’t have to hunt around for external bias (e.g. competitive changes) that might affect LTD.

Think about what I said above about breaking the “average” down into different groups, because this will likely provide the Eureaka! moment and turn the data you are looking at into information.  For example, if you find the LTD of the “average customer”, this is very interesting information indeed, but not highly actionable – what “action” do you take knowing this information?  Can you point to or predict which customers are “average”?

However, if you were to find out the LTD differed dramatically by industry, by salesperson, by country, by time of year, by type of software module installed first – this is highly actionable information, because it provides very direct instruction on where the most profitable areas of business are.

If you lack thoughts in this area, try segmenting by variables that directly affect the experience of becoming a new customer.  At least one of these is most always predictive of LTD, when directly tied to the acquisition of a new customer:

1.  Campaign media (e.g. trade show versus magazine, online versus offline)

2.  Campaign content / offer

3.  Salesperson / Service teams

4.  Product or Category of first purchase

5.  SIC code / Industry (proxy for Product suitability to customer needs)

For example, if you find LTD differs by salesperson, you will find salespeople who create high LTD customers and salespeople who create low LTD customers; the company should study how each salesperson sells and teach the others based on results.

Or perhaps likelihood to purchase again is determined by which customer interface team installed the software – one team does such a good job the customer re-orders the next module very quickly, as compared with other teams.

Knowledge of this type would be extremely valuable to the company – you can use LTD to discover “best practices” hidden within the “average” data, and by spreading those best practices throughout the company, create enormous benefit and increase in profitability.

The secret to creating meaningful customer analysis that delivers high impact is this: always think about what you would **do** with the information you uncover.  If you can’t **do something** with the numbers you evolve, you probably need to drill down a little further and uncover the true meaning of the underlying data.

Hope this helps.  My personal guess based on what I know about behavior would be this: the type of software installed first and the sales / installation / after the sale care team are the two most likely variables predicting speed to next purchase, followed by the industry the buyer comes from.

This is a very interesting project; please keep me informed of your progress and I will help you in any way I can.

Jim

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Segmentation by LTD & LifeCycle

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: I visited your website because I am trying to understand how to develop a customer LifeTime Value model for the company that I work at.  The reason is we are looking at LTV as a way to standardize the ROI measurement of different customer programs.

Not all of these programs are Marketing, some are Service, and some could be considered “Operations”.  But they all touch the customer, so we were thinking changes in customer value might be a common way to measure and compare the success of these programs.

A: Absolutely!  I just answered a question very much like this the other day, it’s great that people are becoming interested in customer value as the cross-enterprise common denominator for understanding success in any customer program!

If I am the CEO, I control dollars I can invest.  How do I decide where budget is best invested if every silo uses different metrics to prove success?  And even worse, different metrics for success within the same silo?

By establishing changes in customer value as the platform for all customer-related programs to be measured against, everyone is on an equal footing and can “fight” fairly for their share of the budget (or testing?) pie.  By using controlled testing, customers can be exposed to different treatments and lift in value can be compared on an apples to apples basis – even if you are comparing the effect of a Marketing Campaign to changes in the Service Center.

But are you sure you want to use LifeTime Value for this application?

Q: From what you stated on your website, I will not be able to develop a LifeTime Value model unless I understand the customer Lifecycle.  The customer lifecycle is something that I could get a good understanding from using doing a RFM analysis.

My question is, once I complete the RFM analysis, what would be my next steps in developing a customer LifeTime Value model?   At this point in time, the hardest thing that I am trying to wrap my head around are the variables to include in the model.  I visited Arthur Middleton Hughes’ website:

http://www.dbmarketing.com

and he suggests the following variables (download spreadsheet, if interested):

http://www.dbmarketing.com/special_ltv.htm

Jim, could I simply use those variables going forward to calculate the LifeTime Value of a customer at my company?  I would appreciate any assistance you may be able to provide to me on this matter.  Thanks.

A: Well, that’s a big tangle of related issues!    Let’s unpack first, then answer the question.  First, the relationships between these ideas:

Lifetime Value versus Lifecycle – LTV is a number, LifeCycle is a trend over time that contains trigger events.  You don’t need the LifeCycle to develop (calculate) LTV, you need the LifeCycle to most efficiently and profitably act on and manage LTV issues.

RFM versus Lifecycle – RFM is a tactical model that is a “snapshot” of customer state at a point in time, the customer’s likelihood to respond.  Frequently used names for these customer states include active, lapsing, lapsed, defected.   Lifecycle is the “movie” one might put together from these snapshots of RFM states; the migration from one customer state to the next are the Lifecycle trigger points.

Now, let’s make sure we understand each one of the ideas:

LifeTime Value

Strictly speaking, LTV is not a very flexible concept and is best used for determining how much you can spend to acquire a customer and still make a profit.  This is the equation that Mr. Hughes has provided, a man by the way that I have a lot of respect for.  His model is quite detailed and useful for the purpose of finding break-even cost to acquire a customer.

To use Arthur’s LTV model, you have to find historical values and plug them in.  You could assume nothing will change and the LTV of certain segments of past customers will be the same; this is great for “benchmarking”, for example.  However, this approach is not measuring LTV, it’s predicting LTV based on historical data.  This is fine, and a valid method for certain types of analysis.

But, the premise of your question is you will be testing, and testing implies something new will occur.  So while you could use LTV to estimate results, you’d have to wait quite a while to prove the results one way or another.  LTV is really “forensic” in this way – you won’t know the final answer until the customers defect.

You could certainly go back 2 – 5 years after the tests, and prove one group had higher LTV than another, but that’s not typically a very useful approach when doing testing.

RFM (Recency, Frequency, Monetary)

RFM is a predictive model that takes a “snapshot” of the customer base and gives you a score for each customer, a prediction of likelihood to respond relative to all customers.

By itself, RFM doesn’t tell you if you are making money or not.  It is used to classify the “state” of customers at a point in time, usually for targeting purposes – are they active, lapsing, lapsed, defected?  In other words, it’s a customer segmentation tool.

For example, RFM could be used to choose your test and control groups for a campaign using Lift measurement – you would want test and control to have the same range and balance of scores.  In fact, one of the tragic campaign measurement mistakes people often make is not taking into account the likelihood to respond when selecting test and control groups, resulting in biased test results.

Customer LifeCycles

One of the great features of RFM is the idea of “ranking” customers relative to each other; this gives allocation of budget and success measurement a standard to follow.  A single  customer can have many different scores over the course of their LifeTime, with the likelihood to respond the score at a specific time.  In fact, if you looked at RFM scores over time for a single customer, you would have a clear understanding of the LifeCycle of a customer – the most powerful segmentation available in terms of message and offer targeting.

The problem with looking at RFM scores over time is complexity; the beauty of individual customer scores at a single point in time becomes unbearable when you are talking 125 different scores on 50,000 customers over 6 months.  That’s the internal or analytical problem.  Externally, this kind of information is extremely gnarly to present and explain to senior managers, it’s presentation hell.

The way I solve this problem is with a tool I call LifeCycle Grids.  The Grids takes the same fundamental drivers used in the RFM model and instead of ranking, uses thresholds or “hurdles” to classify customer states.  This creates a standardized customer LifeCycle “dashboard” so comparisons of customer value between different segments can be made more easily.  It works for both short and long term observations and is easy to represent either numerically or graphically.  And because it uses finite thresholds for activity rather than ranking, the same calculations that create the dashboard can be used to actually drive or trigger actions.

So the dashboard is actually the controller as well.  This is extremely beneficial in terms of linking presentations, plans, and results. People can literally point to a segment on the LifeCycle framework and say, “Let’s deliver message X to each person from segment Y who enters this cell” and see the results right where they pointed when the dashboard is updated.

Once you test some ideas and find out which approach generates incremental profits for a cell in the Grid, you can automate delivery of the program as customers enter that cell of the Grid.  This is the classic “sense & respond” approach to marketing communication – right message, right person, right time.

The LifeCycle Grids are demonstrated in a lot of detail for different applications in the series here, but probably of most interest to you as it relates to customer analysis, see here.

And now, to answer your question:

Which approach above, if any of these, would be best for standardizing measurement of ROI in widely diverse customer programs?

LTV would be appropriate if what you want to know is breakeven cost to acquire.  Since we are talking about customer programs, I doubt that’s what you want to use.  Plus, if you want a hard number rather than a prediction, you could be waiting a long time for the answer.

RFM is a “snapshot” model and so not really suited to long-term studies of customer value.

Customer Lifecycle models are more likely to be involved in the execution of a program, not the success measurement.  LifeCycle tracking could be (and often is) used to predict the financial success of campaigns before they have run their course, but you’re only predicting success, not delivering numbers into an ROI model the CFO would accept as “fact”.

Answer: None of the above.

What you need is an approach designed for the task, which in this case, is:

Lift Measurement or Near-Term Value

Lift is a measure of the performance of a test group of customers compared with a control group of similar customers who are not exposed to the test.  You can read more about control groups here.  In the analysis of value contributed by each group, many of the same values from Arthur’s LTV model are used – product margin, costs of program, fulfillment costs, payment parameters, etc.  However, if you are talking about a program to existing customers, cost to acquire is probably not relevant, though you might use source (campaign) to segment your test approach.

Lift is typically measured at intervals, say every 30 or 60 days, to see how test versus control populations are tracking, and can continue after the test is over to pick up residual value created in the customer.  However, this is not a Lifetime Value measurement, Lift models measure incremental contribution to LTV created by the Marketing, Service, or Operations program execution.

This means if you get lift from program test versus control, when you go back 2 – 5 years later and measure true rather than predicted LTV – after the customer has defected – you should in fact see the LTV in the test group higher than in the control group, barring any radical downstream difference in customer experience between test and control.  In this way, Lift models are actually predictive of changes in LTV.  That’s why the output of Lift models is sometimes referred to as the measurement of “Near-Term Value” and used much more often than the forensic approach of waiting for customers to defect.

Summary

All the above are core concepts in customer value measurement and management.

LTV is a measurement of net financial value contributed by a customer, and Lift measures  are like a “time slice” of the overall LTV curve.

LifeCycles are a management framework for programs designed to affect LTV, and models using Recency, Frequency, and Monetary are used to look at a “time slice” of the LifeCycle.

LTV can generally be increased in two ways: by creating more value during the existing LifeCycle, or by extending the LifeCycle.  Marketing (including Product) is typically used when doing the first, Service and Operations – customer experience and satisfaction – are largely what affects the second.

So it is completely appropriate to establish a unified approach to the measurement of customer programs intended to increase the value of a customer across all these disciplines, in order to ensure the allocation of  scarce resources to highest and best use.

A great question, and for a great cause!

Jim

Update:

Listrak asked me to do a podcast with them on these and related topics, check it out (MP3 link) here, or see list of all their Email Marketing Today podcasts here (I’m on Episode 42).

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

LTV, RFM, LifeCycles – the Framework

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: Thank you for your recent article about Control Groups.  Our organization launched an online distance learning program this past August, and I’ve just completed some student behavior analysis for this past semester.

Using weekly RF-Scores based on Recently and Frequently they’ve logged in to courses within the previous three weeks, I’m able to assess their “Risk Level”– how likely they are to stop using the program.  We had a percentage who discontinued the program, but in retrospect, their login behavior and changes in their login behavior gave strong indication they were having trouble before they completely stopped using it.

A: Fantastic!  I have spoken with numerous online educators about this application of Recency – Frequency modeling, as well online research subscriptions, a similar behavioral model.  All reported great results predicting student / subscriber defection rates.

Q: I’m preparing to propose a program for the upcoming semester where we contact students by email and / or phone when their login behavior gives indication that they’re having trouble.  My hope is that by proactively contacting these students, we can resolve issues or provide assistance before things escalate to the point they defect completely.

A: Absolutely, the yield (% students / revenue retained) on a project like this should be excellent.  Plus, you will end up learning a lot about “why”, which will lead to better executions of the “potential dropout” program the more you test it.

Q: However, in light of your newsletter, I realized that we should probably have a control group with whom we do NOTHING (just as we did this past semester) in order to prove the effectiveness (or not) of the program.

A: Correct.  Otherwise, you won’t be able to make a valid claim to the “saved students”. People can always argue a variety of other factors were in play – seasonality, topic, course sequence, etc.

Q: Since the actual number of students is confidential, can you please tell me what percentage you would use for a control group if we had 400, 800, 1200, 1600, 2000, 3500, or 5000 students?  You mentioned 10% in your newsletter, but the population you were referring to exceeded millions.

A: Well, there are online calculators you can use confidentially, example right here.

If you don’t understand the variables they are asking for, explanations at bottom of page, though this is very simple – what is confidence level and interval plus population size.

Q: Our population is MUCH smaller, and each customer is therefore even more critical.  I don’t want to recommend an unnecessarily large control group that would prevent us from retaining future students when we could see they were having trouble.

I suspect that our defection rates will be lower 2nd semester than 1st since students should be beyond the “learning curve,” so I don’t think we can justly say that the program alone is the reason for lower defection rates if we don’t use a control group.

A: Yes, well, this desire to “get as much test as we can” was the main point discussed in the newsletter.  And that’s the challenge with very small populations – to hit statistical confidence levels at say population = 500, you need over 300 or so in control.

Not so great.

So we go back to the question of company culture and how intuitively confident people will be with the results.  Do they in fact need true statistical significance for a program like this?

There is a way around the significance issue – repetition. The stats part of this is all about the “likelihood you get the same results again” – real important for drug testing, not so much for 500 folks in a marketing program.

The question you need to ask: do you really need “prediction”?  Or does prediction just make the whole test more complex and expensive than it’s worth?  What if you repeated the test a couple of times and got roughly the same results, is that “proof”?

Here is what I might do.  I would ask whoever needs to believe in the results of this test a question like this:

“Let’s say we took a random 20% sample of the students and excluded them from the marketing.  We apply the marketing to the other 80% and their retention rate is 15% higher than the 20% who had no marketing. We do this test 2 more times and the retention rate of students in the test is 13% and 17% higher than the students in the 20% who do not receive the marketing.  Would you at that point believe that without question, the marketing drives at least a 13% improvement in retention among students?”

Do you see where I’m headed with this?  The more times you repeat the test, the more confident you will be in the results – regardless of sample sizes and statistical mumbo jumbo. At some point, the reality of the differences between test and control performance has to be accepted.  It may help to define up front how many repetitions the “boss” needs.

There are two clues to help you evaluate the validity of your results / how many times you need to repeat the test to be “confident”.

One clue is the variability of the results – the more inconsistent the results are, the more likely the data is “noisy” and the more times you need to repeat the test to be confident.

If the spreads between test and control for the first 3 tests are 20%, 5%, and 10%, then you’ll need more repetitions of the test to get a good feeling for the actual impact.  If the results tend to cluster as in the example above (15%, 13%, 17%) then you can be more confident earlier in the test series the actual impact is somewhere around 15%.

The other clue is in the “spread” between test and control.  If the spread is consistently  ”wide”, say +10% (or more), this provides additional confidence a positive impact is being made.  The result over a series of tests may not actually be +10% (confirm by repeating the test), but it’s more likely to be positive.  If you consistently get a spread more like 1% or 2%, it’s more likely the actual result could be zero or negative and you need to keep repeating the test to gain confidence you have a positive result.

In the end, you may not want or be able to repeat the test enough times to know with statistical confidence what the result is.  But if the spread between test and control is wide and consistent, and the cost relative to the benefit is small, then does it really matter if there is statistical confidence?

For example, if you can make the statement you’re confident the program generates at least $10 in profit for each $1 invested, does it really matter if the statistically confident  number is $11 or $12 profit for $1 in cost?  We’re doing Marketing here, not drug testing.  There is an opportunity cost (profit left on the table) to not rolling out a program based on a test with results like this; rather than repeat the test to death just to be more confident I’d roll it out and continue to monitor the results.

One more tip, on this idea of sequencing / semesters / experience with the program.

There is no doubt in my mind that 2nd semester students would have what is called a “survivor bias” and be less likely to drop out; you will get the best performance in a program like this with 1st semester students.  So if at all possible, run the test / control on only 1st semester students , or segment by semester.

But, just because you run it on only 1st semester students does not mean you don’t have an effect in 2nd semester.  Continue to follow test and control into 2nd, 3rd, 4th semesters and you may see the dropout rate of the original 1st semester group continue to widen versus control.

This is not only great for the profitability of the initial 1st semester program but also provides you the baseline you have to beat (control) for those 2nd, 3rd, 4th semesters.  When you decide to see if you can have an additional effect by intervening in those periods, you’ll have 2 groups: those affected by Marketing in the 1st semester, and those new to any Marketing intervention.

My guess: a 1st semester intervention will have tremendous impact, both then and throughout the 4th.  The impact of intervention at each subsequent semester will diminish compared with acting in 1st semester, as will the “tail” value created over the student life, since the number of months left in the student life is shrinking each semester.

Hope that helps!

Jim

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Control Groups in Small Populations

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

 Q:  I am a big fan of your web site and read your Drilling Down book. Great work!

A:  Thanks for the kind words!

Q:  I was wondering if you could help me picking the right control group size for a project of ours?  The population is 25 million telco customers that for which we want to do a long term impact analysis (month by month) in regards to revenue increase versus control group.  The marketing initiatives are mix of retention, lifecycle and tactical/seasonal activities.  We want to measure revenue increase through any of the marketing activities compared to control group.

A:   Great project, this is the kind of idea that can really improve margins if you can find out which specific tactics drop the most profit to the bottom line.

Q:   I have searched the web for some help and found calculators that say: On 25 million and smallest expected uplift of 0.1% and highest likely rate of > 5% the calculator gives 250k (1%).  Is that sufficient to calculate the net impact on the remaining base?  Would be very grateful if you could give me your thoughts.

A:  Well, it could be and might not be…

If you were manufacturing widgets, where the outcomes are clear (unit is defective or not defective), you might use this approach to the question.  But in Marketing we’re talking about human behavior, and there is quite a lot more variability in outcomes and more room for interpretation.  You can encounter a number of problems down the road by running a control so “tight” to the statistically correct size.

From a practical perspective, when you do a test of this magnitude (and I assume strategic importance), you don’t want test to just “beat control”, you want to beat control beyond a shadow of any executive’s possible doubt.

From personal experience, I can tell you that executives tend to be non-believers with a 1% control versus a 5% control or a 10% control. So some of this control size choice is culture-based – if the exec team is a bunch of engineers that understand / believe in statistical sampling methods, then 1% is probably OK in terms of believing the results are predictive of future events.

But if you need to convince a CFO or somebody who will be working from gut or risk management rather than “science” then 1% may not be enough, there is too much perceived “room for error” with a 1% sample (even with the science).

This is in effect a “perceived confidence interval” argument – the difference between 95% confidence and 99.999% confidence. Engineers may be OK with 95% because they intimately understand the derivation of it; CFO’s not so much.  CFO’s may not even understand the math behind confidence but intuitively, they perceive that 10% control is “more likely to be accurate” than 1%.

Said another way, do you want people to argue about the math and stats and waver on their belief in the outcome, or do you want them to just look at a simple chart of test versus control numbers and say, “Congratulations, that’s a tremendous success!”.  A 10% control gets you complete agreement on the results without any quibbling.  At 1%, you may get “what about the chance we are wrong” arguments.

Now, there are financial implications to using very large controls – some positive (reduced expense), and some negative (potential revenue foregone).  So choosing control group size can be impacted by these other issues.  In small population tests these financial impacts are usually quite small, so negligible and I always go for large controls.

But in a population of 25 million, maybe not so.

Which brings us to the second consideration -  segmentation or “drill down” after the test.

Nothing is quite so painful as gearing up for a test of this magnitude, producing a stunning positive result on a “macro” basis across all initiatives, and then having the execs ask, “What is the driving force behind this increased profitability in the test group?  Is it retention, lifecycle or tactical / seasonal?”  Or as often happens in telco (usually from an ops GM or VP), “What was the result of this test in my region or on my platform?”

Uggghh…

With a 1% control across the entire population, you frequently are “boxed in” when it comes to sub-populations because you lose significance (both perceived and scientific) as you drill in.  You may be OK on a couple of large scale events on large populations, but as we know, every answer begs another question and you can run out of statistically significant answers pretty quickly.  If you use a large control at the macro level, you are (as a rough example) 99% confident at the macro level, 98% confident one segment down, 97% confident two segments down, 95% confident three segments down, etc.

One way to handle this is to build the test from subsegments up to the macro level.  Let’s say at a minimum you want 3 subsegments in the test – retention, lifecycle or tactical / seasonal – and each of these you want to be 95% confident in.  Since some of these programs are triggered by behavior (lifecycle) and some by calendar (seasonal) I’d guess the sizes of the populations and number of executions could be vastly different.  Meaning, you may only need 1% control on the seasonal promotions but more like 5% or 10% control for some of the lifecycle stuff to be 95% confident on the outcomes of those.

When you sum all these segments up, you often end up with more like 2% or 3% of the entire population in control groups to always be at least 95% confident at all the desired subsegments, which means you end up with even higher confidence at the macro “all campaigns” level – a very good thing.

And much better than trying to explain why you can’t answer a subsegment question because you used 250K instead of 400K or 600K in the control group, if you know what I mean!  That’s when people forget the arguments about foregone revenue and start saying stuff like “Why did you not use a larger control group for this test?”

In the end, you will thank yourself again and again for using a larger than minimum required control at the macro level because you WILL come up with that unexpected “must know” question and be thrilled to find out you actually can answer it at a decent level of confidence.

Good luck with it, let me know what you learn!

Jim

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Choosing the Size of Control Groups

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: You kindly clarified a few issues when I was reading Drilling Down earlier this year – so I hope you don’t mind the direct email.

A: Yes, I remember!

I am working for www.XYZ.com, a social networking / virtual world site based abroad but visitors are 85% US.

Our growth up to now has been mainly viral and in the summer we hit 1.2M UVs operating on the Freemium model with only 5% of our registered users converting to paying customers and a significant portion of our revenue coming from ads.  On average our customers are active on the site for something like 4 months making their first purchase around day 28. 

But to take us to the next stage we are embarking on some marketing for the first time using AdWords and various revenue share campaigns, and of course to do this sensibly we need to arrive at a reasonable estimate of LTV.

A: Makes sense!

Q: To calculate an adjusted LTV I removed all customers with a lifetime of less than 4 months but this gives a low estimate as this calculation ignores the bumper summer months and the extra paid for features put in place earlier this year.  Calculating LTV using ARPU and monthly churn (not sure how to calculate this in our environment) gives another different estimate.  Is there any help or advice you could perhaps give us?  If not in the US then perhaps you could recommend somebody abroad – can’t find anything in the literature relevant for start-up like us.

A:  It sounds to me like you’re trying to make this too complicated, at least for the place you are at this time.  Monthly churn and the “28 day” threshold are nice to know on a tactical level, but LTV is more of a Strategic idea that does not necessarily benefit from analysis at that level.  And you may not really want LTV, but a derivative that might be more helpful.

Let’s say the average user sticks around 4 months.  Say also that you generate revenues of $1 million over that period, and 1,000,000 users had some level of activity.  So your revs per active user are $1.  In terms of generating net revenue, you want to acquire  users for less than $1.

Now, we know that number is topline, and obviously there are expenses.  Companies like yours do not have very straightforward financial models because of the large amount of R & D that may be capitalized rather than expensed.  So you need to go to someone in Finance and determine what the right number is to use for looking at ROI.  Ask them, what percent of our revenues are left over to pay bills?  Or, what number would you like to see increased through a Marketing program?

Is it cash flow? Earnings before Interest, Taxes, Depreciation, Amortization?  Gross Margin?  Some other?  Then, what percent of sales does this number generally run at?

Let’s say it’s 40%.  In other words, 40% of revenues is actually available to pay bills and so forth.  So in the example above, .4 x $1 = 40 cents, which is the max you can pay to acquire a user, and anything less than that generates money to pay bills.

This method of course looks at all revenues. Not sure why you would want to look at it any differently, since even users that don’t “purchase” still generate ad revenues.

But let’s say you want to be more specific, you care only about buyers and only want to run campaigns that generate buyers.  In other words, the advertising revenue is “nice to have” but you want to build out the paid marketing model based on the acquisition of visitors likely to purchase.  You can run the same model above, but only look at the known the buyer group.

Take any 4 month period, find revenues from purchases and divide by number of people purchasing (not purchases, but individuals who purchase, revenue / user).  Then apply the same 40% flow through from the model above, and that’s the max you can pay to acquire a buyer.

When you aggregate known buyers, segment by source and you will find different campaigns generate different kinds of buyers; some will stick around longer than 4 months, some less.

Segment these folks by campaign and run the same model as above, purchase revenue for the period they stick around divided by purchases times the 40% flow through.  That’s the max you can pay to acquire a buyer in that segment.  So you end up with (just guessing) being able to pay 5 cents for campaigns that generate people who stick around 2 months, 15 cents for people who stick for 3 months, and 40 cents for people who stick 4 months.

You can make LTV equations very complex, but often the point of the exercise is not really “what is the customer LTV?” it’s “how much should we spend to improve cash flow?” or something similar.  This is a much easier question to answer and often what the company really wants to know.

Said another way, it’s very difficult and often dangerous to peg an LTV number in a dynamic environment because there are so many potential changes that will impact it; LTV is a number you may fully understand 2 – 5 years from now.  Until then, you need something “close” that drives the same kind of thinking and action, and the approach above will get this done for you.

As things evolve, this number (called “flow through”, recently have heard it called “Near Term Value”) will basically approach true LTV as you extend the number of months in the measurement period.

At the point where your business is somewhat static at an operational level, you can then look for true LTV by examining the revenue of actual defectors.  This is the only way to really peg LTV – after users have left and sufficient time has elapsed where you do not believe they will come back by themselves.

Until then, you are better off trying to figure out how much you can pay to attract high  quality user / buyer segments.

Jim

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Customer Value in the Freemium Model