Has to be There

I find it really interesting that whenever there is a discussion of measuring the value of social media, there’s such a bias towards believing there is value in social beyond what can be properly measured.  See the comments following this post by Avinash for a good example.  Speculation is fine, but the confidence being expressed that a new tool or method will uncover a treasure trove of social media value seems un-scientific (as in scientific method) at best.

I don’t doubt there is some value in social media beyond what can be measured, as this has been the case for all marketing since marketing measurement began.  These measurement problems are not new to social either:  Marketing value created is often situational, it depends on the business model and environment.  What works in one situation may not work in another.

For example:

To small local businesses who do no other form of advertising, there is a huge amount of relative value to using social media, versus no advertising at all.  Social advertising is much better than none, and since it’s free, the incremental value created by (properly) using social is huge.  It’s also really easy to measure the impact and true value, since the baseline control is “no advertising”.  Lift, or actual net marketing performance, can be pretty obvious in his case.

On the other hand, many companies are running lots of advertising designed to create awareness, and the incremental value of social as a “media” may be close to zero for these companies, or at least less than the cost to analyze the true value of it.  Possible explanation:  Social events such as “Likes” or comments are simply representations or affirmations of awareness already created by other media, so by themselves, create little value.  In other words, events such as Likes might track the value of other media spending, but may not create much additional marketing value.

Why is this plausible?  It seems unlikely a person would “Like” a brand or product if they have not already experienced it, and are already a fan.  This means in the vast majority of cases, little incremental awareness / acquisition is created.  If this case is not true, if people “Like” a company even though they have no reason to (paid to Like?), then the problems with social marketing analysis go way beyond tools – the concept and data driving the analysis itself is flawed.

But if Like really means Like, the number of Likes or any other similar social events do not have as much to do with awareness as they do with the size of a loyal customer base, and are much more aligned with tracking the success of other awareness / acquisition campaigns.

Looking for Love in All the Wrong Places?

That all said, I believe there is some value being created in the acquisition / awareness area from social.  The problem seems to be this value, when measured, is quite a bit less than everyone expects.  So “the hunt for social value” seems never ending, with speculation and measurements contrived from thin air immensely  popular.  This missing value just has to be there, right?

The core problem is an old one: online value measurement definitions are all over the map, so it’s easy to claim value was created by simply inventing a new way to measure success.  I can’t wait for the day when established test and measurement standards (like using control groups) are adopted in the online space.

Meanwhile, I think it’s quite possible if people keep looking in the awareness / acquisition area, the value of social “beyond what can now be measured”, in many cases, is probably less than the cost of actually measuring it.

Why this approach?  Based on my experience, People are Social, Media are not.  So if you want to derive social value, you use people metrics, not media metrics.

Using this approach, I have unbridled optimism for the value of social.

But I won’t go as far as insisting value is there without measuring it properly first.  Because that’s not how science works.

See ya at eMetrics NYC!

P.S.  There’s lots of real experimental science out there on the effects of social media in the marketing space, have you reviewed it?

You will find this material to be a treasure trove of new ideas and proper methods worth pursuing in the social measurement space, examples here, here, here, here.  Get yourself a subscription to Marketing Science or if you are a WAA member, you can request copies of these fully documented social measurement experiments.

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

“Missing” Social Media Value

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: I purchased your book and have a few questions you can hopefully help me out with.

A: Thanks for that, and sure!

Q: We have 4 product lines and 2 of them are seasonal. i.e we have customers that year in year out purchase these items consistently but seasonally, for example, every spring and summer.  Then they are dormant for Fall and Winter.  Should I include these customers along with everyone else when doing an RFM segmentation?

A: Well, it kind of depends what you will using the RF(M) model for, what kinds of marketing programs will be activated by using the scores. If you know you have seasonal customers and their habit is to buy each year, AND you wish to aim retention or reactivation programs at them, I would be tempted to divide the customer base so that seasonal customers are their own segment.  Then run two RF(M)  models – one for the seasonals, and one for everyone else.

Q: If I include seasonal customers, and I run RFM say on a monthly basis, these seasonal customers will climb / fall drastically with time depending on the season, so it seems like it may complicate the scoring process.

A: Sure, and you could segment as I said above.  Or, you could run across a longer time frame, say across 2 – 3 years worth of data. This would “normalize” the two segments into one and take account of the seasonality in the scoring – perhaps be more representative of the business model.  However, the scores would become less sensitive due to the long time frame so the actions of customers less accurately predicted by the model.

Q: Can you provide me with some examples as to how segmentation is carried out?  Let’s say I being with RFM and all my customers are rated 5-5, 5-4, 4-5 etc.  What are the next steps, do we overlay with other characteristics like age, gender, etc?  Or are the 5-3 etc. our actual segments?

A: This goes back to what you want to use the RF(M) model for.  In the standard usage, each score will have roughly the same number of customers in it, those with higher scores will be more likely to respond to marketing and purchase, lower scores less likely.

Another way to say this is: since the lower scores are less responsive, they require higher value discounts or promotions to activate them.  So you can customize offers based on score, which will provide the maximum response for the minimum discount exposure.

Try testing different discount levels with different scores to see where you get maximum profit; an example of that kind of testing and math is here:

Using Recency to Drive Promotional Profit

The above assumes you are focused on response and profit, two of the more common objectives in data-based marketing ;)

However, other valuable information can be discovered using this framework, see chapter 20 about adding customer characteristics to RF(M) scores.  This is not needed if you are just concerned about response / profit for campaigns – the segments are the segments.

But if you’d like to know, for example, what kind of merchandise is appealing to 5-5 (ultra-best, highest responding) customers, you could run the scoring, take the 5-5’s, and then cross-tab with whatever else you have – find out what kind of merchandise they buy, what their sex is, etc. – any kind of data you have or can get on them.  This kind of work can help develop creative, decide what to feature on the home page, etc.

Q: Should we provide them a segment name like High Recent / Low Frequents, Low Frequents / Low Recents etc?  How do the other characteristics come into play with respect to the naming conventions, segmentation etc.  I keep reading of different segments like ‘Loyalists’, ‘Laggers’, are these defined through our subjectivity?

A: Generally yes, subjective labels; you may  find it is easier to communicate using labels rather than scores, so you could suggest that certain ranges of scores be called something like ‘Loyalists’ or ‘Laggers’.

The High Recent / Low Frequents segmentation is the quadrant approach from LifeCycle mapping, see the chapter 22 in the book.  This is an alternative way to use the RF(M) variables to map your customer base in a more Strategic way than RF(M) affords.

In other words, you could use RF(M) to score for Campaigns and the LifeCycle maps to report on your progress over time to management, since both approaches are based on the same variables – great way to “connect the dots” for manic-ment.  Also, RF(M) is a bit hard to visualize just using the scores; the LifeCycle maps allow you to plot campaign results in a very visual, easy to understand display of the data.

See these newsletters for a deeper explanation of Strategic versus Tactical use of RF scoring:

RFM versus LifeCycle Grids

LTV, RFM, LifeCycles – the Framework

Hope that helps!

Jim

===================

Q: How do I establish the optimal number of segments / buckets I should have when analyzing a customer database?  I was using a method of dividing them into deciles and assigning Decile 1 to Segment A, Decile 2 & 3 to Segment B, Deciles 4 – 6 to Segment c and Deciles 7 – 10 to Segment D.  However, the absolute number in each is not practical as I will not be able to manage that number of customers in the way I need to.

A: Well, it would help to know what Kind of business this is, but in general, the most effective place to break behavioral segments is where behavior changes.  You can force people into Deciles, which often does not create meaningful segments, or you can look at the data and see where there are “bulges” or changes that seem significant.

So, for example, look at this LifeCycle Grid:

Current / Potential Value Matrix

Note the low Current Value breaks of 1, 2, 3 because there are large groups of customers there.  Then, 4 – 9, because each of those has many fewer customers and their behavior is similar.  Same with 10 – 24, that’s a pretty good customer comparatively, and after 25 purchases, well, they’re crazy-good customers and behave similarly.

These cutoffs start out as somewhat arbitrary, but over time and testing you find out, for example, that a 9X buyer behaves more like a 5X buyer and not so much like a 10 – 24X buyer, so the cutoff is 9X.

Here’s another example:

Freemium Customer Conversion

The segments graphed in this post are actually deciles of 1 year customer value; but the data plot is of the first 14 weeks of weekly spending.  Note how there are clear inflection points where the  behavior changes, and this is where I would generally create the “boundaries” for a segment.  Doing it this way means the segmentation already has some type of powerful behavioral trait, making it by definition significant and “actionable”.

Hope that helps!

Jim

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Defining Behavioral Segments

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: We’ve been playing around with Recency / Frequency scoring in our customer email campaigns as described in your book.  To start, we’re targeting best customers who have stopped interacting with us.  I have just completed a piece of analysis that shows after one of these targeted emails:

1. Purchasers increased 22.9%
2. Transactions increased 69%
3. Revenue increased 71%

A: There you go!

Q: My concern is that what I am seeing is merely a seasonal effect – our revenue peaks in July and August.  So what I should have done is use a control group as you described in the book – which is what I am doing for the October Email.

A: Yep, that’s exactly what control groups are for – to strain out the noise of seasonality, other promotions, etc.  But don’t beat yourself up over it, nothing wrong with poking around and trying to figure out where the levers are first.

Q: Two questions:

1.  What statistical test do I use to demonstrate that the observed changes are not down to chance

2.  How big should my control group be – typically our cohort is 500-800 individuals

A: Good questions…

On a group that small, you are probably not going to get anything “statistically significant” without ruining your total profit, e.g. might have to use 50% in control.   If you have the leeway to do it, that’s what I would do.

On the other hand, in some cultures people will go bonkers over giving up sales to learn something really important.  OK, so take 10% as control and repeat it 3 times; if the results are stable then you have your proof.  Do another control every once and a while (every 6 months?) just to make sure it tracks.

Either way, you don’t really need statistics.

Practically, confidence is the likelihood a sample represents the population.  This can be a really useful idea when you are forced into very small sample sizes or the event is highly risky to repeat.  But here, if you are testing a really large slug of the population, confidence is less useful.  Or if you can repeat the event (because essentially, you are in control of it and it’s low risk), do you really need to force yourself through the screw of complying with the statistical math?  It’s like using a 727 for crop dusting, overkill for the situation, methinks.

If you were running a drug manufacturing line, statistical concepts like confidence and significance are absolutely valuable.  But for a marketing program?

That’s why I love the idea of “beefy controls” in start-up projects because I *do not* have to rely on statistics that the audience likely does not understand and provide room to question the results, e.g. “Yea, but what if the result is an outlier?”  Very appropriate in high risk situations, with giant populations and a lot of money on the line.  For this situation, perhaps not.  But, if you’d like to go that way, there’s lots of calculators on the web that let you play with some of the numbers anyway.

Here’s one, make sure to read the descriptions of the variables underneath the calculator:

http://www.surveysystem.com/sscalc.htm

Nice work on the core campaign idea, by the way!  Now we just have to tighten it up a bit…

(3 months later)

Q: We decided to tighten the targets and do a “best customer defection” email program. Basically, we look at customers who has an RFM score of 555 in the previous scoring period who have dropped out of that score.

A: Interesting!  So instead of targeting by  guessing the current score of a defecting best customer (say 355), you are looking for all customers who were formally best customers, regardless of current score.  This is a subtle difference, but much more of a LifeCycle approach and frankly why I prefer these kinds of ideas over “straight” RFM.

An example might be helpful.  Let’s say the acquisition folks run a huge new customer campaign in between the prior RFM scoring and the scoring done before your campaign drop.  A big inflow of new customers can artificially “force” certain groups of customers down in score – even though their own behavior has *not changed*.  In this case, the new score is not reflective of actual behavior, so increases noise in the system.

That’s the problem with the “Snapshot” or date-specific view of Customer State – it’s a single point without reference.  By using prior score, you are acknowledging behavior over time and the primary importance of the former State, as opposed to the current State – a Movie as opposed to a Snapshot.

In other words, from a  Marketing perspective, I’m more interested in the path they are taking through the LifeCycle than any particular point in time during the LifeCycle represented by a single RFM score.

Q: Good news on your advice.  We ran a 50% control (500 purchasers in each group) and the results really nailed the issue for us. The actual number of purchasers remained unchanged at 20% but Total Revenue and Average Spend increased by 40% compared to control.

(Jim’s Note: for those not following, a very precise target group of 1000 was split into 2 groups of 500.  One group received this  campaign, the other did not.  People who did not receive the campaign purchased at the same rate as people who did receive the campaign, but the people who received the campaign averaged 40% higher spend).

A: Awesome.  So what you are seeing is Customer State makes a huge difference in terms of what offers / timing can be most effective for this “Recently defecting best customers” cohort.  If I’m reading your numbers correctly, no lift in response versus control but a huge lift in revenue.

To me, that means these customers are early in the process of defection – still buying, but without a special treatment, slowing down the monthly spend.  After all, they are very Recent (former 5XX), so highly likely to purchase again, which is why lift in response was flat – they likely would have purchased anyway.

Not a bad time to hit them.  Offers to a very Recent State should focus on increasing order value, not generating response – you don’t want to spit into the wind, but go with the natural flow of the behavior.

In other words, these customers likely would have purchased anyway, but at lower price points if they had not received the campaign.  The common way this is addressed is with  “threshold” discounts – if average order is $50, then something like “$10 off any purchase over $50″ – test different thresholds to maximize profitability.

Looks like you gave them the right offer ;)

On the other hand, a straight discount to this specific best customer group – $10 off anything, and especially when their normal category of purchase is promoted to them – almost ensures that you will lose money.  Why?  Most of these customers would have bought at full price anyway, as demonstrated by equal buying activity whether the customer received the campaign or not.  So the discount turns into a loss versus no campaign at all.

Unfortunately, I see a lot of this exact type of campaign delivered to best customers because all customers get some version of the same offer.  “Hey Jim, we’re not sending the same message to every customer, we send different messages by segment”.  Sure, the copy and art are customized for different segments, but the segmentation is not by Customer State, so the offers are mismatched and suboptimal.

This is the value of using control groups; they drive understanding of Marketing concepts like opportunity costs and subsidy costs.  These two concepts are the reasons why ignoring Customer State is suboptimal: by not segmenting using State, you will get lower than possible profit or sales at most customers, depending on Customer State.

Had you not delivered a campaign tailored for prior Customer State, money would have been left on the table by way of lower order size.  And 40% Revenue lift sounds like it might have covered the cost of the campaign ;)

Q: We tried to run a Student T test on the results but our new statistician informed me that the distributions were not normal – so on her advice we ran a Wilcoxan Test which gave us a highly significantly p = 0.016

A: Oh, so you still went the stats route?  Well, the fact you HAVE a statistician tells me the culture there is more familiar with interpreting these ideas, so more power to you.

Glad it worked out and keep me informed on how things go downstream.

Jim

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Increase Profit Using Customer State

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: Thank you for creating this useful website!

A: You’re welcome!

Q: When figuring out retention rate for an annual or a 8 months life time cycle period, how do I pick the starting period?  Do I look at their first orders on a date?  Or I pick a time frame such as one month?

A: It depends on:

1. What kind of “retention” you are talking about, the definition, which is probably impacted by the audience for the data

2.  What you will do with the retention data, what kind of decisions will be made and actions be taken because of the data

You should always ask these questions above  when someone requests “retention data” – or any other kind of analysis, for that matter!  For example, there probably is a huge difference in what you would provide to the Board of Directors for an annual benchmark and what you would provide to Marketing people for executing campaigns.

In the first case, the data would probably be used to inform Strategic decision making, for example, should we change our product mix or approach to pricing given the market?  In the second case, the data would probably be used in a Tactical way, for example, to target new customers who are predicted to defect because of the campaign they responded to or the product they bought.

If providing data to the Board, “annual retention rate” would probably make the most sense (again, you should ask, what’s it for?). If that’s what they want, you would pick a starting period, probably aligned with the fiscal year (Jan – Dec?), and find out what percent of people who purchased Jan – Dec 2009 also purchased Jan – Dec 2010.

That’s the annual retention rate.  Useful information, perhaps leading to the Board requesting action of some kind.  But by itself, you really can’t “do” anything with this data, there’s no source or targeting information, there’s no customer value information.

However, if you segment by campaigns, product of initial purchase, price points, offers, or other actionable variables, the retention rate could be just about any formula, e.g. what is the retention rate:

a. Today, of people who made their first purchase in 2005?
b. End of 2009, of people who made their first purchase in 2005?
c. Today, of people who ever bought Product X as their first purchase?
d. Today, of people who bought Product X as their first purchase in 2009?
e. Today, of people who had at least 2 service calls in 2010, who became new customers in 2009, who used a 50% off promotion?

and so on.  Retention rate for anything tactical almost always requires and audience and time frame to be defined.

Q: You mention in your article, “Total number of customers” as the denominator for calculating the customer retention rate, do you mean the total customers at the end of the period?  Or those total customers came in on the first date of a fixed period?  Or the first fixed period that I’m observing?

A: Whatever definition is the correct definition depending on the need of the audience.  There is no standard, other than perhaps the very first one, the Strategic “reporting” idea of year over year retention.  This is commonly used in reporting to Wall Street, for example.

While discussing this particular idea of “customers”, one might encounter the common problem of not knowing the definition of a customer, at least in terms of retention.

When does the company declare a customer is no longer a customer?  Is a customer  “everyone” who has ever purchased?  If the company has been around 10 years, and you are calculating retention rate “today”, as in how many of these total customers purchased in the last year, you may find you have a very low number, one that won’t mean much to anybody, and is not actionable.

On the other hand, if your definition of “customer” includes a level of activity, for example, “must purchase at least twice, one of those purchases in the past 3 years”, now you are talking about a highly actionable kind of retention definition.  Why?

Because there is some hope that people who have purchased at least 2x (Frequency), at least once in the past 3 years (Recency) could actually still be customers, as opposed to defected customers.  If I am calculating a “serious” retention rate, something to be used to take Marketing action, or pay out bonuses, or revise policies, I want to measure against people who actually have some Potential Value, some Value to the company in the future.  That’s how I define a customer.  To me, there isn’t any point in calling someone a customer who is unlikely to contribute to profits in the future.

If you define as a customer “anyone who purchased over the past 10 years”, you just have a dead metric that really does not reflect the reality of what taking action might produce.  In other words, you are including people who are extremely unlikely to still be customers, so what’s the point of the “customer retention metric” you created?

Does the above help answer your question?

Q: I wasn’t expecting you to reply me so fast and in such detail!!!  Thank you so much!  I’m calculating this retention rate for marketing and your answer is very helpful for me!!!

A: Great!  So maybe ask them specifically how they want to look at it, and if they seem puzzled, suggest to them various options.

I can tell you from experience with businesses like yours is the buying behavior tends to peak early and you have to act quickly if you want to extend the lifecycle.  Perhaps not quite as time-critical given your “triple bottom line”, but probably not too different.

This argues for a tighter leash on the definition of a customer, perhaps purchased at least twice, one of those past 6 months.  You could also do 2x purchase, at least once in past 3 years, and compare, it will give them a feel for customer defection trend / rate.

The next step would be the Lifecycle map, which uses Recency and Frequency in a more actionable way, like this example.

Marketing people should be able to use this map to target specific groups of customers, e.g. purchased 4 – 9 times, but not in the past 90 days.  These are good customers who are in the process of defecting, and require special attention to keep them on board.

After all, the point of measuring retention is not retention rate itself, it’s about increasing the productivity and profitability of the business system.  Just as you can optimize for conversion, you can optimize for retention, and sometimes you discover they conflict.

For example, one company I worked with featured certain products on their home page because those products had a high conversion rate on visits to the home page; they had “optimized” the home page for this scenario.

However, a very quick and simple calculation showed these products generated customers  with terrible repeat purchase rates relative to just about every other product with volume.   A quick survey of these customers found out why the repeat purchase rates were so low – almost all customers disliked the product and thought the company deceived them.  Turns out the company “over-sold” the product – and that’s why the high conversion rates.

In another case, PPC campaigns had been optimized for conversion without regard to customer retention.  Under a budget crunch, the lowest converting campaigns were killed, but overall sales volume over the next 3 months dropped much more than the sales generated by these campaigns.

Reason?  These low converting campaigns generated the company’s very best customers in terms of 30-day, 90-day, 180-day value, while most of the highest converting campaigns generated low value, single purchase customers on the same time frames.

This kind of analysis is simply not that difficult to set up and execute, relative to the extreme amounts of value that can be created:

1.  Pass campaign codes / info with the order to the backend order processing.  If you are not doing this yet, start right now!

2.  Select a campaign, choose a time frame.  If you want to match up to financial statements (a good idea if you will be talking to C-Level folks), say January 2010.

3.  Grab all new customers who came in on Campaign X during Month Y – what is their average value 1, 3, 6, 12 months later?  This is a Lifecycle by Campaign analysis, similar to the LifeCycle map example mentioned above.

The new customer experience (channel, offer, product) is one of the most powerful predictors of future customer value, and the value of these new customers relative to each other tends to remain stable regardless of how many other generic campaigns (weekly email) you throw at the customer over time.

Across all campaigns, about 60 – 80% of these new customers will have the same value at 12 months they had at 1 month.  The question to answer, as with any optimization, is this: knowing the customer value created by these campaigns varies widely, are we allocating the acquisition spend optimally?  For example, are we spending 70% of the budget to generate  20% of the annual customer value?  Are we willing to pay more for clicks that generate new customers with 10X higher annual value?

Retention rate isn’t just some mystical number, retention rate quickly turns into profit dollars and can have incredible financial impact!

Jim

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Optimizing for Customer Value

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: I was wondering if you’ve done any work with, or given thought to, companies who have a cloud based Freemium business model?

Should they be tracking usage (or anything) at the free level?  Should they be tracking usage at the paid level?  I’m sure defection rates are a big problem, but I’m wondering how many focus on engagement thru mass marketing versus trying to keep what they’ve got, or influence the free users to make the leap to paid.  Any thoughts on this?  Maybe you could do a blog post on it.  It seems like a good fit with your brand of analysis but I’m just starting to think it through…

A: I just finished an analysis that’s a good example of this problem.  Behavior during the Freemium period can predict who is highly likely to become a paying customer, who will need marketing efforts like additional sampling / package discounts, and who will not become a customer no matter what you do.

So the answer is you need both, analysis of paid and free.  But in particular, what you need to do is understand the transition from free to paid by comparing the behavior of known converters versus non-converters over time, preferably using events that create value for customers.

Typically the differences will be volume / persistence related, generically, low Recency high Frequency.  Also, likely converters to paid will tend to use a wider variety of features consistently.  In the analysis, the question to answer is which of these value-creating behaviors is predictive of becoming paid?

Said another way, you tend to see a fairly fast drop-off among *all* new Freemium customers after the initial burst of activity, but the ones that are not going to stick tend to drop off even faster in their activity.  Then, there is a “bounce” in activity where the ones who are most likely to end up as paid begin to cycle behavior more quickly and begin to use more features, and the others simply drift off the map, with no “bounce” as Recency becomes extended.

Classic LifeCycle analysis – customers tell you what they will become in the future by what they do today.  Having the very detailed behavioral information typically seen with interactivity just multiplies the ability to do this kind of prediction.  More on the Freemium model, including determining appropriate cost to acquire, is here.

Q: Do the standard analytics packages allow a business to look back at the “free” behavior of paid subscribers?  I’m thinking of Freemium cloud based solutions and how they would track this.  Do products like Crazy Egg get you there or do you really need something more sophisticated to do this kind of analysis?

A: I’ve never used Crazy Egg so I don’t know about that one specifically, but in general you can do quite a lot with the basic tools that support customizable segmentation.  The challenge with going that way is you have to be super-technical with the implementation to capture important event data points, you have to create many different segments, and then the killer problem – you can’t “re-analyze” a different approach with these tools, for the most part.  If that’s what you mean by “look back”, it’s highly unlikely you could accomplish what you need to do.

So it’s possible, but these tools are not really designed for “behavior over time” work and certainly don’t allow for much “exploration” of the data – any change in analytical approach is likely to be a “going forward” type of measurement, not looking back.  So there would be lots of iteration before you even knew if the analytics set-up was correct or what events are meaningful.  In other words, it’s possible but could waste a LOT of time.

I’d much rather find the system that contains the key elements of activity – when did they sign up, what features are they signed up for, when did they add other features.  This data probably resides in whatever system manages the account.  Dump that data off into a spreadsheet or database, try to figure out what’s meaningful, look for correlations.

Then, once you have a grip on some solid ideas, then maybe you go into the front end and try to align traffic and behavior with known “events” that seem to predict upgrade to pay, if that’s what the mission is.

Otherwise, you will be setting up a ton of tracking on all kinds of events not knowing what is meaningful, and then dealing with a really poor interface for the analysis of those events.

The other way to go, of course, is to use one of the advanced web analytics tools, which sit on real databases and can be queried.  But assuming that’s not an option, I would try to look for hard data points in the backend first, then knowing key behaviors, look for what might cause those behaviors in the traffic side.

Example

Let’s say you have a project management application that has a 60-day free trial then converts to paid.  Value is created for the customer when they use the functionality of the app – say create project, comment on project, upload file, or any other actions you deem critical.  “Traffic” in a situation like this may be only marginally indicative of value creation; rising activity could even be a negative indicator (frustration with application).

So, you want to create a situation where you can analyze these important behavioral events by account, and (ideally) you want to know the source of the account creation – campaign code, referrer, etc. That’s all you need for data, simple table, maybe a dozen columns.

Then, compare average account that converts to paid with average account opened at the same time as the converters but does not convert, over the 60-days before trial end.  For each of converting and non-converting, aggregate each of key events by week, divide by number of accounts to get average behavior per account, and you would have 8 weekly average data points for each of the events, both for non-converting and the converting accounts.  Maybe a dozen simple line graphs with 8 weekly data points, one set for accounts that paid, one set for accounts that did not.

Converting and non-converting graphs should look different for some variables.  Both will typically start out with high levels of activity, then for some variables you will see them diverge.  This not only predicts which variables affect conversion, but reveals to you the best time during the 60-days to intervene with surveys, help, or other marketing programs to re-engage the accounts that appear to be headed for defection.  If you have campaign data, also which campaigns tend to create accounts that convert and which don’t.

One of the event graphs may look to be more predictive than the others, with abrupt changes in direction going into the conversion event.  For example, perhaps it will look like this:

(Click pic for larger image)

This is the behavior of 10 different 1st year spend levels (deciles) over the first 14 weeks of their Life, engaging in an event that creates value for them.  The dark blue line represents average top spender.  Note how for top spenders, the profile is quite different.  The graph tells you that by week 4 or so, you can probably predict who will become a best customer and who will need intervention based on this activity.

You can run this kind of event profile for any variable – events, campaigns, etc. as long as you know complete / non-complete goal or end value of the customer.  In your case, since the goal outcome is binary, there would be 2 lines instead of the 10 spending deciles above: converters versus non-converters.  Create a converter versus non-converter chart for each key activity variable (create project, comment on project, upload file, or any other actions you deem critical), and look for this kind of divergence.

Drilling down more deeply by excluding all but 3 lines so we can see the behavior “in the middle”, we find some interesting patterns:

(Click pic for larger image)

Here, we see patterns that provide clues to the testing targets one might want to address to see if “middle” customers could be turned into better customers.  The blue segment, showing a series of higher highs and higher lows after it “bottoms” for this behavior, is most likely to benefit from intervention of some kind.  The pink segment looked promising, but then put in lower highs and lower lows – these customers lose momentum quickly and have trouble self-sustaining.  The yellow segment was never really in the game at all.

Yes, the comparison to stock market charting is intentional!  It’s an expression of group behavior.

If I had to pick the segment with the best potential, I would try the blue segment first, and the data points could be used for automated triggering of different types of campaigns. For example, “If by week 4 activity for Variable X  falls below 60, trigger Campaign A.  Then if by week 11 activity for Variable X is not above 40, trigger Campaign B.”  Remember, these are averages, so not all customers in the segment are below threshold.  The idea is to target a specific behavior with a specific message.

Just to be clear, you don’t need the goal value of the customer to put a model into practice, only to prove the initial model – certain patterns in behavior predict high value customers.  Once you know the end value of  customers – convert or not, monetary value, any goal – you can run the LifeCycle movie “backwards” like the charts above and find out which early  behaviors are predictive of high and low value customers.

If you want to go further, you could show these graphs and data to a modeler and see if they can create a more precise mathematical model, which can be developed much more quickly with this kind of evidence to review.

Once you fully understand what this LifeCycle landscape looks like, THEN you could go back and instrument the web site and analytical tool to monitor some version of this data in a more automated way.  But trying to guess what’s going to be important beforehand and work through a study like this using a vanilla web analytics tool is the very, very long way to get where you need to go!

Jim

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Freemium Customer Conversion

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: One of the first things I am doing in my new job is to identify the Customer Lifecycle pattern – how many periods (month, year) will it be before a customer is likely buy again.  In enterprise software industry, where software cost easily 6 figures, # of years is a reasonable time frame.

A: Yes, one would assume this.  But these notions would most likely be based on a feeling of the “average” behavior, and on average, it probably does take a long time.

What is not known is this:  if the “average” is composed of short-cycle and long-cycle buyers, who are the short cycle buyers, and what are they like?  What industry SIC code, for example?  And can we get more of them, or at least focus more resources on them, if they are the most profitable?  So the challenge is not only to look for the “average”, but then understand how this average is composed.  If you can break down the average by industry, or by salesperson, for example, this might be highly directional information.

Q: From my internal analysis, however, I discerned from the sales figures something quite counterintuitive – the period between first and next sale is much shorter than I would have thought for the SW industry in general.

A: Pleasant surprise, eh?  I don’t know what kind of figures you are looking at, but make sure the data is in fact what you think it is.  For example, if you want to study software purchase itself, do the “sales” figures you are looking at include transactions involving not the sale of software, but also service, like installation or modification fees?  It would make sense that a “software sale” would be followed pretty quickly by an “installation” sale, for example.  You need to know this to properly segment.

Q: Would you be able to point me to some studies on how often customers wait after the first purchase before contemplating an upgrade of software or something you personally have done in consulting projects for SW companies?  This industry benchmark will then shed some light on whether this trend is something peculiar to our company or not…

A: I am not aware of any published study of this type.  And as you might imagine, these numbers would vary quite widely in the industry and the nature of the information would be a highly guarded corporate secret.  So I don’t think you will find any “benchmark” studies of this type, and sorry, I can’t share client data with you!

Q: The next step for me then is to map out the drivers for this behaviour and then calculate the LTV (LifeTime Value) and take a look at the actual LifeCycle events creating this LTV.

A: Yes, but to be precise, LTV is typically a forecast when working with current customers; it’s not known until the customer actually defects, marking end of  “Life”.  What you are probably looking for is more accurately called “Life to Date” (LTD), the actual sales of a customer from start of relationship until present.

Also, when segmenting customers by LTD, of course look for temporal bias – a 10-year customer is likely to have higher LTD than a 2-year customer.  If there are enough customers, it might be a good idea to first segment by start year, then LTD.  This way you have cohorts of customers who are going through the same experiences together and differences in LTD will be more significant in terms of predictive power – you don’t have to hunt around for external bias (e.g. competitive changes) that might affect LTD.

Think about what I said above about breaking the “average” down into different groups, because this will likely provide the Eureaka! moment and turn the data you are looking at into information.  For example, if you find the LTD of the “average customer”, this is very interesting information indeed, but not highly actionable – what “action” do you take knowing this information?  Can you point to or predict which customers are “average”?

However, if you were to find out the LTD differed dramatically by industry, by salesperson, by country, by time of year, by type of software module installed first – this is highly actionable information, because it provides very direct instruction on where the most profitable areas of business are.

If you lack thoughts in this area, try segmenting by variables that directly affect the experience of becoming a new customer.  At least one of these is most always predictive of LTD, when directly tied to the acquisition of a new customer:

1.  Campaign media (e.g. trade show versus magazine, online versus offline)

2.  Campaign content / offer

3.  Salesperson / Service teams

4.  Product or Category of first purchase

5.  SIC code / Industry (proxy for Product suitability to customer needs)

For example, if you find LTD differs by salesperson, you will find salespeople who create high LTD customers and salespeople who create low LTD customers; the company should study how each salesperson sells and teach the others based on results.

Or perhaps likelihood to purchase again is determined by which customer interface team installed the software – one team does such a good job the customer re-orders the next module very quickly, as compared with other teams.

Knowledge of this type would be extremely valuable to the company – you can use LTD to discover “best practices” hidden within the “average” data, and by spreading those best practices throughout the company, create enormous benefit and increase in profitability.

The secret to creating meaningful customer analysis that delivers high impact is this: always think about what you would **do** with the information you uncover.  If you can’t **do something** with the numbers you evolve, you probably need to drill down a little further and uncover the true meaning of the underlying data.

Hope this helps.  My personal guess based on what I know about behavior would be this: the type of software installed first and the sales / installation / after the sale care team are the two most likely variables predicting speed to next purchase, followed by the industry the buyer comes from.

This is a very interesting project; please keep me informed of your progress and I will help you in any way I can.

Jim

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Segmentation by LTD & LifeCycle

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: I visited your website because I am trying to understand how to develop a customer LifeTime Value model for the company that I work at.  The reason is we are looking at LTV as a way to standardize the ROI measurement of different customer programs.

Not all of these programs are Marketing, some are Service, and some could be considered “Operations”.  But they all touch the customer, so we were thinking changes in customer value might be a common way to measure and compare the success of these programs.

A: Absolutely!  I just answered a question very much like this the other day, it’s great that people are becoming interested in customer value as the cross-enterprise common denominator for understanding success in any customer program!

If I am the CEO, I control dollars I can invest.  How do I decide where budget is best invested if every silo uses different metrics to prove success?  And even worse, different metrics for success within the same silo?

By establishing changes in customer value as the platform for all customer-related programs to be measured against, everyone is on an equal footing and can “fight” fairly for their share of the budget (or testing?) pie.  By using controlled testing, customers can be exposed to different treatments and lift in value can be compared on an apples to apples basis – even if you are comparing the effect of a Marketing Campaign to changes in the Service Center.

But are you sure you want to use LifeTime Value for this application?

Q: From what you stated on your website, I will not be able to develop a LifeTime Value model unless I understand the customer Lifecycle.  The customer lifecycle is something that I could get a good understanding from using doing a RFM analysis.

My question is, once I complete the RFM analysis, what would be my next steps in developing a customer LifeTime Value model?   At this point in time, the hardest thing that I am trying to wrap my head around are the variables to include in the model.  I visited Arthur Middleton Hughes’ website:

http://www.dbmarketing.com

and he suggests the following variables (download spreadsheet, if interested):

http://www.dbmarketing.com/special_ltv.htm

Jim, could I simply use those variables going forward to calculate the LifeTime Value of a customer at my company?  I would appreciate any assistance you may be able to provide to me on this matter.  Thanks.

A: Well, that’s a big tangle of related issues!    Let’s unpack first, then answer the question.  First, the relationships between these ideas:

Lifetime Value versus Lifecycle – LTV is a number, LifeCycle is a trend over time that contains trigger events.  You don’t need the LifeCycle to develop (calculate) LTV, you need the LifeCycle to most efficiently and profitably act on and manage LTV issues.

RFM versus Lifecycle – RFM is a tactical model that is a “snapshot” of customer state at a point in time, the customer’s likelihood to respond.  Frequently used names for these customer states include active, lapsing, lapsed, defected.   Lifecycle is the “movie” one might put together from these snapshots of RFM states; the migration from one customer state to the next are the Lifecycle trigger points.

Now, let’s make sure we understand each one of the ideas:

LifeTime Value

Strictly speaking, LTV is not a very flexible concept and is best used for determining how much you can spend to acquire a customer and still make a profit.  This is the equation that Mr. Hughes has provided, a man by the way that I have a lot of respect for.  His model is quite detailed and useful for the purpose of finding break-even cost to acquire a customer.

To use Arthur’s LTV model, you have to find historical values and plug them in.  You could assume nothing will change and the LTV of certain segments of past customers will be the same; this is great for “benchmarking”, for example.  However, this approach is not measuring LTV, it’s predicting LTV based on historical data.  This is fine, and a valid method for certain types of analysis.

But, the premise of your question is you will be testing, and testing implies something new will occur.  So while you could use LTV to estimate results, you’d have to wait quite a while to prove the results one way or another.  LTV is really “forensic” in this way – you won’t know the final answer until the customers defect.

You could certainly go back 2 – 5 years after the tests, and prove one group had higher LTV than another, but that’s not typically a very useful approach when doing testing.

RFM (Recency, Frequency, Monetary)

RFM is a predictive model that takes a “snapshot” of the customer base and gives you a score for each customer, a prediction of likelihood to respond relative to all customers.

By itself, RFM doesn’t tell you if you are making money or not.  It is used to classify the “state” of customers at a point in time, usually for targeting purposes – are they active, lapsing, lapsed, defected?  In other words, it’s a customer segmentation tool.

For example, RFM could be used to choose your test and control groups for a campaign using Lift measurement – you would want test and control to have the same range and balance of scores.  In fact, one of the tragic campaign measurement mistakes people often make is not taking into account the likelihood to respond when selecting test and control groups, resulting in biased test results.

Customer LifeCycles

One of the great features of RFM is the idea of “ranking” customers relative to each other; this gives allocation of budget and success measurement a standard to follow.  A single  customer can have many different scores over the course of their LifeTime, with the likelihood to respond the score at a specific time.  In fact, if you looked at RFM scores over time for a single customer, you would have a clear understanding of the LifeCycle of a customer – the most powerful segmentation available in terms of message and offer targeting.

The problem with looking at RFM scores over time is complexity; the beauty of individual customer scores at a single point in time becomes unbearable when you are talking 125 different scores on 50,000 customers over 6 months.  That’s the internal or analytical problem.  Externally, this kind of information is extremely gnarly to present and explain to senior managers, it’s presentation hell.

The way I solve this problem is with a tool I call LifeCycle Grids.  The Grids takes the same fundamental drivers used in the RFM model and instead of ranking, uses thresholds or “hurdles” to classify customer states.  This creates a standardized customer LifeCycle “dashboard” so comparisons of customer value between different segments can be made more easily.  It works for both short and long term observations and is easy to represent either numerically or graphically.  And because it uses finite thresholds for activity rather than ranking, the same calculations that create the dashboard can be used to actually drive or trigger actions.

So the dashboard is actually the controller as well.  This is extremely beneficial in terms of linking presentations, plans, and results. People can literally point to a segment on the LifeCycle framework and say, “Let’s deliver message X to each person from segment Y who enters this cell” and see the results right where they pointed when the dashboard is updated.

Once you test some ideas and find out which approach generates incremental profits for a cell in the Grid, you can automate delivery of the program as customers enter that cell of the Grid.  This is the classic “sense & respond” approach to marketing communication – right message, right person, right time.

The LifeCycle Grids are demonstrated in a lot of detail for different applications in the series here, but probably of most interest to you as it relates to customer analysis, see here.

And now, to answer your question:

Which approach above, if any of these, would be best for standardizing measurement of ROI in widely diverse customer programs?

LTV would be appropriate if what you want to know is breakeven cost to acquire.  Since we are talking about customer programs, I doubt that’s what you want to use.  Plus, if you want a hard number rather than a prediction, you could be waiting a long time for the answer.

RFM is a “snapshot” model and so not really suited to long-term studies of customer value.

Customer Lifecycle models are more likely to be involved in the execution of a program, not the success measurement.  LifeCycle tracking could be (and often is) used to predict the financial success of campaigns before they have run their course, but you’re only predicting success, not delivering numbers into an ROI model the CFO would accept as “fact”.

Answer: None of the above.

What you need is an approach designed for the task, which in this case, is:

Lift Measurement or Near-Term Value

Lift is a measure of the performance of a test group of customers compared with a control group of similar customers who are not exposed to the test.  You can read more about control groups here.  In the analysis of value contributed by each group, many of the same values from Arthur’s LTV model are used – product margin, costs of program, fulfillment costs, payment parameters, etc.  However, if you are talking about a program to existing customers, cost to acquire is probably not relevant, though you might use source (campaign) to segment your test approach.

Lift is typically measured at intervals, say every 30 or 60 days, to see how test versus control populations are tracking, and can continue after the test is over to pick up residual value created in the customer.  However, this is not a Lifetime Value measurement, Lift models measure incremental contribution to LTV created by the Marketing, Service, or Operations program execution.

This means if you get lift from program test versus control, when you go back 2 – 5 years later and measure true rather than predicted LTV – after the customer has defected – you should in fact see the LTV in the test group higher than in the control group, barring any radical downstream difference in customer experience between test and control.  In this way, Lift models are actually predictive of changes in LTV.  That’s why the output of Lift models is sometimes referred to as the measurement of “Near-Term Value” and used much more often than the forensic approach of waiting for customers to defect.

Summary

All the above are core concepts in customer value measurement and management.

LTV is a measurement of net financial value contributed by a customer, and Lift measures  are like a “time slice” of the overall LTV curve.

LifeCycles are a management framework for programs designed to affect LTV, and models using Recency, Frequency, and Monetary are used to look at a “time slice” of the LifeCycle.

LTV can generally be increased in two ways: by creating more value during the existing LifeCycle, or by extending the LifeCycle.  Marketing (including Product) is typically used when doing the first, Service and Operations – customer experience and satisfaction – are largely what affects the second.

So it is completely appropriate to establish a unified approach to the measurement of customer programs intended to increase the value of a customer across all these disciplines, in order to ensure the allocation of  scarce resources to highest and best use.

A great question, and for a great cause!

Jim

Update:

Listrak asked me to do a podcast with them on these and related topics, check it out (MP3 link) here, or see list of all their Email Marketing Today podcasts here (I’m on Episode 42).

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

LTV, RFM, LifeCycles – the Framework

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: Thank you for your recent article about Control Groups.  Our organization launched an online distance learning program this past August, and I’ve just completed some student behavior analysis for this past semester.

Using weekly RF-Scores based on Recently and Frequently they’ve logged in to courses within the previous three weeks, I’m able to assess their “Risk Level”– how likely they are to stop using the program.  We had a percentage who discontinued the program, but in retrospect, their login behavior and changes in their login behavior gave strong indication they were having trouble before they completely stopped using it.

A: Fantastic!  I have spoken with numerous online educators about this application of Recency – Frequency modeling, as well online research subscriptions, a similar behavioral model.  All reported great results predicting student / subscriber defection rates.

Q: I’m preparing to propose a program for the upcoming semester where we contact students by email and / or phone when their login behavior gives indication that they’re having trouble.  My hope is that by proactively contacting these students, we can resolve issues or provide assistance before things escalate to the point they defect completely.

A: Absolutely, the yield (% students / revenue retained) on a project like this should be excellent.  Plus, you will end up learning a lot about “why”, which will lead to better executions of the “potential dropout” program the more you test it.

Q: However, in light of your newsletter, I realized that we should probably have a control group with whom we do NOTHING (just as we did this past semester) in order to prove the effectiveness (or not) of the program.

A: Correct.  Otherwise, you won’t be able to make a valid claim to the “saved students”. People can always argue a variety of other factors were in play – seasonality, topic, course sequence, etc.

Q: Since the actual number of students is confidential, can you please tell me what percentage you would use for a control group if we had 400, 800, 1200, 1600, 2000, 3500, or 5000 students?  You mentioned 10% in your newsletter, but the population you were referring to exceeded millions.

A: Well, there are online calculators you can use confidentially, example right here.

If you don’t understand the variables they are asking for, explanations at bottom of page, though this is very simple – what is confidence level and interval plus population size.

Q: Our population is MUCH smaller, and each customer is therefore even more critical.  I don’t want to recommend an unnecessarily large control group that would prevent us from retaining future students when we could see they were having trouble.

I suspect that our defection rates will be lower 2nd semester than 1st since students should be beyond the “learning curve,” so I don’t think we can justly say that the program alone is the reason for lower defection rates if we don’t use a control group.

A: Yes, well, this desire to “get as much test as we can” was the main point discussed in the newsletter.  And that’s the challenge with very small populations – to hit statistical confidence levels at say population = 500, you need over 300 or so in control.

Not so great.

So we go back to the question of company culture and how intuitively confident people will be with the results.  Do they in fact need true statistical significance for a program like this?

There is a way around the significance issue – repetition. The stats part of this is all about the “likelihood you get the same results again” – real important for drug testing, not so much for 500 folks in a marketing program.

The question you need to ask: do you really need “prediction”?  Or does prediction just make the whole test more complex and expensive than it’s worth?  What if you repeated the test a couple of times and got roughly the same results, is that “proof”?

Here is what I might do.  I would ask whoever needs to believe in the results of this test a question like this:

“Let’s say we took a random 20% sample of the students and excluded them from the marketing.  We apply the marketing to the other 80% and their retention rate is 15% higher than the 20% who had no marketing. We do this test 2 more times and the retention rate of students in the test is 13% and 17% higher than the students in the 20% who do not receive the marketing.  Would you at that point believe that without question, the marketing drives at least a 13% improvement in retention among students?”

Do you see where I’m headed with this?  The more times you repeat the test, the more confident you will be in the results – regardless of sample sizes and statistical mumbo jumbo. At some point, the reality of the differences between test and control performance has to be accepted.  It may help to define up front how many repetitions the “boss” needs.

There are two clues to help you evaluate the validity of your results / how many times you need to repeat the test to be “confident”.

One clue is the variability of the results – the more inconsistent the results are, the more likely the data is “noisy” and the more times you need to repeat the test to be confident.

If the spreads between test and control for the first 3 tests are 20%, 5%, and 10%, then you’ll need more repetitions of the test to get a good feeling for the actual impact.  If the results tend to cluster as in the example above (15%, 13%, 17%) then you can be more confident earlier in the test series the actual impact is somewhere around 15%.

The other clue is in the “spread” between test and control.  If the spread is consistently  ”wide”, say +10% (or more), this provides additional confidence a positive impact is being made.  The result over a series of tests may not actually be +10% (confirm by repeating the test), but it’s more likely to be positive.  If you consistently get a spread more like 1% or 2%, it’s more likely the actual result could be zero or negative and you need to keep repeating the test to gain confidence you have a positive result.

In the end, you may not want or be able to repeat the test enough times to know with statistical confidence what the result is.  But if the spread between test and control is wide and consistent, and the cost relative to the benefit is small, then does it really matter if there is statistical confidence?

For example, if you can make the statement you’re confident the program generates at least $10 in profit for each $1 invested, does it really matter if the statistically confident  number is $11 or $12 profit for $1 in cost?  We’re doing Marketing here, not drug testing.  There is an opportunity cost (profit left on the table) to not rolling out a program based on a test with results like this; rather than repeat the test to death just to be more confident I’d roll it out and continue to monitor the results.

One more tip, on this idea of sequencing / semesters / experience with the program.

There is no doubt in my mind that 2nd semester students would have what is called a “survivor bias” and be less likely to drop out; you will get the best performance in a program like this with 1st semester students.  So if at all possible, run the test / control on only 1st semester students , or segment by semester.

But, just because you run it on only 1st semester students does not mean you don’t have an effect in 2nd semester.  Continue to follow test and control into 2nd, 3rd, 4th semesters and you may see the dropout rate of the original 1st semester group continue to widen versus control.

This is not only great for the profitability of the initial 1st semester program but also provides you the baseline you have to beat (control) for those 2nd, 3rd, 4th semesters.  When you decide to see if you can have an additional effect by intervening in those periods, you’ll have 2 groups: those affected by Marketing in the 1st semester, and those new to any Marketing intervention.

My guess: a 1st semester intervention will have tremendous impact, both then and throughout the 4th.  The impact of intervention at each subsequent semester will diminish compared with acting in 1st semester, as will the “tail” value created over the student life, since the number of months left in the student life is shrinking each semester.

Hope that helps!

Jim

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Control Groups in Small Populations

Which brings me to the latest Marketing Science article I have reviewed for the Web Analytics Association, Dynamic Customer Management and the Value of One-to-One Marketing, where the researchers find “customized promotions yield large increases in revenue and profits relative to uniform promotion policies”.  And what variable is most effective when customizing promotions?

The researchers took 56 weeks of purchase behavior from an online store, and used the first 50 weeks to construct a predictive model of purchase behavior.   Inputs to the model included Price, presence of Banner Ads, 3 types of promotions, order sizes, number of orders, merchandise category, demographics, and weeks since last purchase (Recency).

The last 6 weeks of data were used to test the predictive power of the model, and the answer to which variable is most predictive of purchase is displayed in the chart below, click to enlarge:

Weeks since last purchase dominated the predictive power of the model, controlling not only the Natural purchase rate (labeled Baseline in chart above, people who received no promotions) but the response to all three different types of promotion.

The  Natural buying rate (here, as much as 50% of campaign response) has tremendous implications for the measurement of Campaign profitability, and can also be used to measure the success of customer-centricity / experience / social programs.  These are the issues I cover in my review of the article.  If you’re interested in that take, you can read it here.

But for this post, what I’d like to do is  explore the Recency measurement idea itself, because I suspect a lot of people may not understand what it really means.  And since many Marketing folks are not used to taking action on this kind of data, also talk about what you can do with this information.

Most people think of time in a linear way.  A graph that includes time typically starts at some point in the past and churns through time in a sequential fashion.  Not so with the graph above, which is looking at Purchase Cycles.

In this style of cycle measurement, customers are moved back to the time = zero segment (left side of chart) as soon as a purchase is made, and time starts all over again for these customers.  If they don’t make a purchase, they continue to slide out down the curves to the right.  Can you picture this activity in your mind?

You can have customers who stay at the top end of the graph, rapidly cycling round back to zero weeks each time they purchase.  You can have customers with longer cycles that loop back to zero weeks in slower purchase cycles from the middle.  You can have customers who purchase only once and every week just slide out further way from zero until they fall off at the right.

That means the same person might be in different  places on a curve in different weeks.  The same person can buy 2 weeks after last purchase or 4 weeks after last purchase, or a person can buy every week, or every month.  All of these purchase cycles summarized produce the series of likelihoods you see on the chart.

The point of the chart is, no matter which promotion customers are exposed to, no matter when their previous purchase was made (2 weeks ago or 20 weeks ago), their likelihood to purchase again can be very simply and accurately predicted by knowing one simple data point: weeks since last purchase.

Said another way, because this is a core concept to customization using behavior:

Customers with all kinds of different purchase patterns, demographics, categories of purchase, campaign exposure, and so forth tend to behave in the same way, that is, their likelihood to purchase at any given point in time from this  online store is primarily a function of how long it’s been since they last purchased from the store.

There are some pretty significant online marketing implications from a statement like that.  But how do you act on this information?

You’ve probably heard of the “sales pipeline” idea from B2B.  Sales management gathers data to inform them on which deals are likely to close and when, and build a flow chart of expected revenues.  This helps management take action on any deals that seem to be “floundering” -  special exec attention, discounts, bundling, etc.

You can do this in B2B because the value of the customers is usually quite high, and you have sales people or account managers who are close to the customer and can provide this data.

In B2C, you can’t afford to have account people for each customer, but using Recency you can predict which groups of customers are most likely to purchase again, and then build the same kind of sales pipeline.  And then, customize your Marketing action based on whether the customer seems likely to buy or is ”floundering” and drive increased profitability.

Building a sales pipeline model can also be used to predict how well the business will be doing in the future, and what kinds of products or tactics are really driving future profits.  Like other kinds of optimization, moving focus or resources towards products and tactics that are driving value, and away from those destroying it, results in a more profitable business.  But using Recency, instead of optimizing the Present, you are really optimizing the Future.

Look at the chart above.  There is a discount promotion and a free shipping promotion.  The coupon promotion outperforms the free shipping promotion as long as the customer has purchased in the past 6 weeks.  After this point, free shipping outperforms coupons.  That is something, as a Marketer, I think I’d like to know.  It means to optimize this system, I should deliver campaigns not based on my calendar, but based on the customer’s calendar as evidenced by their purchase cycle behavior.

Similarly, around week 8 since last purchase, coupon performance drops below the baseline performance of people in the loyalty program.  And finally, at 20 weeks, coupon performance is basically equal to the Natural buying rate, meaning virtually everyone using a coupon would have purchased anyway without the coupon.

Please understand, I’m not saying these Recency curves will be the same for your commerce site – they will depend on the type of products you sell, how good your service is, and so forth.  You have to do your own analysis.  What I am saying is the Recency effect is universal and can be the most important variable you could ever use for segmentation if you are concerned about campaign profitability.

For a practical perspective however, data in the format above is difficult to use and explain to other folks.  I much prefer what I call the LifeCycle Grid format below, click to enlarge:

People are more used to seeing data in a format where “up and to the right = better” so I have flipped the zero Recency boundary to the right side.  The customers with the lowest future value are in the lower left (Pink) and highest future value are in the upper right (Green).  I have also cross-tabbed Recency with Frequency so we have an idea of the value of a customer; the value of the customer helps decide how to approach the customer.   For Recency, we have chosen “hard breaks” rather than a smooth curve.  This creates specific populations so we can target certain groups and measure results.

Example:  If I send a 10% off promotion to all customers, you will see dramatic differences in response and profitability across these different cells.  Working the grid this way with various offers, you will find that allocating the same Marketing budget and promotions evenly across all the cells is truly a suboptimal approach.

Additionally, the general location of the cell gives clues to customizing campaign content or angle of attack as well as customizing the offers.  In general, for the four colored segments:

Green: Best customers who are Engaged – this is a segment where aspirational messages and services are extremely effective.  Think ”Special VIP treatment” in campaign copy and offers.

Orange: Best customers with declining likelihood to purchase again – if you are truly customer-centric, it’s time to analyze (or survey) these customers for broken products, processes, and service.  Why is a best customer dis-engaging?  Can we help you?  Did we do something wrong?  Would you recommend us?

Yellow: Potential Best Customers – new customers and those who are “floundering”.  What can you do to turn them on?  This is a group that benefits from category or affinity analysis to inform campaign content; help them try new product ideas.

Pink: Defected Low Value Customers - high value, broad discounting (30% off anything) is probably the only thing that’s going to drive response from this group – is it really worth it / do you actually generate profits here?

From a management perspective, feeding specific populations through the Grids can inform strategic decisions.  If you believe the Grids essentially represent a sales pipeline, then how do the pipelines for different customer segmentations compare?

An obvious place to start is Campaigns – what do the sales pipelines look like for different Campaigns, which Campaigns generate the highest percentage Green segment 1 month after Campaign drop?  What about at the end of month 3?

Run Product or Category analysis through the Grids.  For example, new customers whose first purchase is in a certain category – does this category create customers with high pipeline value?  What about customers who continue to buy in the category?  Softgoods versus hard goods?  Software versus hardware?  Shouldn’t we feature products that drive high pipeline value in campaigns and on the home page, as opposed to products that generate 1x buyers?

How about channel analysis, which sources generate new customers with the highest likelihood to continue purchasing?  Are most of our PPC customers in the Green segment, and most of our Affiliate customers in the Pink segment?  Where do the Social customers end up?  At 1 month after first purchase?  At the end of month 3?

The beauty of this approach is it can be used over and over, on any platform, in just about any situation, to answer the same question: which activities generate customers with the highest future value?  The Grids provides a consistent way to compare investments in all types of activities – products, campaigns, service initiatives, usability, centricity.  Just take the population exposed to the test, run them through the Grid, and compare to average (or better yet, control).

Most Marketers grew up with a linear view of execution – just keep Pushing, the more impressions the better.  Taking this approach in an Interactive environment completely ignores the fact that many customers will come back and Purchase again without any Push at all - and especially so if you are nailing all the centricity angles.

The trick is to optimizing Interactive commerce for Profit is:

1.  Understand which tactics create customers with high pipeline value – those likely to re-purchase on their own - then,

2.  Take Marketing action based not on a linear calendar, but a cyclical one – the calendar defined by the customer’s own behavior, customizing the message by location of the customer in the purchase likelihood Grid.

Execution Tips: List selection for this customization program is easily automated, right?  Just use the Grid cell boundaries as selection variables.  Many people decide to keep a regular generic “Brand” email communication to all customers while running the hyper-targeted communications based on cycle behavior underneath.  In this case, consider backing off discounting in the Brand communication and stick to new products, new hires, content marketing, etc. and let the cycle-driven email handle the behavioral discount program.  Test for the optimal balance / frequency between the 2 different emails by tagging e-mails with Grid cell.

Questions on this?  Also, with this background you might now want to read my review of the study.

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Acting on Buyer Engagement

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

 Q:  I am a big fan of your web site and read your Drilling Down book. Great work!

A:  Thanks for the kind words!

Q:  I was wondering if you could help me picking the right control group size for a project of ours?  The population is 25 million telco customers that for which we want to do a long term impact analysis (month by month) in regards to revenue increase versus control group.  The marketing initiatives are mix of retention, lifecycle and tactical/seasonal activities.  We want to measure revenue increase through any of the marketing activities compared to control group.

A:   Great project, this is the kind of idea that can really improve margins if you can find out which specific tactics drop the most profit to the bottom line.

Q:   I have searched the web for some help and found calculators that say: On 25 million and smallest expected uplift of 0.1% and highest likely rate of > 5% the calculator gives 250k (1%).  Is that sufficient to calculate the net impact on the remaining base?  Would be very grateful if you could give me your thoughts.

A:  Well, it could be and might not be…

If you were manufacturing widgets, where the outcomes are clear (unit is defective or not defective), you might use this approach to the question.  But in Marketing we’re talking about human behavior, and there is quite a lot more variability in outcomes and more room for interpretation.  You can encounter a number of problems down the road by running a control so “tight” to the statistically correct size.

From a practical perspective, when you do a test of this magnitude (and I assume strategic importance), you don’t want test to just “beat control”, you want to beat control beyond a shadow of any executive’s possible doubt.

From personal experience, I can tell you that executives tend to be non-believers with a 1% control versus a 5% control or a 10% control. So some of this control size choice is culture-based – if the exec team is a bunch of engineers that understand / believe in statistical sampling methods, then 1% is probably OK in terms of believing the results are predictive of future events.

But if you need to convince a CFO or somebody who will be working from gut or risk management rather than “science” then 1% may not be enough, there is too much perceived “room for error” with a 1% sample (even with the science).

This is in effect a “perceived confidence interval” argument – the difference between 95% confidence and 99.999% confidence. Engineers may be OK with 95% because they intimately understand the derivation of it; CFO’s not so much.  CFO’s may not even understand the math behind confidence but intuitively, they perceive that 10% control is “more likely to be accurate” than 1%.

Said another way, do you want people to argue about the math and stats and waver on their belief in the outcome, or do you want them to just look at a simple chart of test versus control numbers and say, “Congratulations, that’s a tremendous success!”.  A 10% control gets you complete agreement on the results without any quibbling.  At 1%, you may get “what about the chance we are wrong” arguments.

Now, there are financial implications to using very large controls – some positive (reduced expense), and some negative (potential revenue foregone).  So choosing control group size can be impacted by these other issues.  In small population tests these financial impacts are usually quite small, so negligible and I always go for large controls.

But in a population of 25 million, maybe not so.

Which brings us to the second consideration -  segmentation or “drill down” after the test.

Nothing is quite so painful as gearing up for a test of this magnitude, producing a stunning positive result on a “macro” basis across all initiatives, and then having the execs ask, “What is the driving force behind this increased profitability in the test group?  Is it retention, lifecycle or tactical / seasonal?”  Or as often happens in telco (usually from an ops GM or VP), “What was the result of this test in my region or on my platform?”

Uggghh…

With a 1% control across the entire population, you frequently are “boxed in” when it comes to sub-populations because you lose significance (both perceived and scientific) as you drill in.  You may be OK on a couple of large scale events on large populations, but as we know, every answer begs another question and you can run out of statistically significant answers pretty quickly.  If you use a large control at the macro level, you are (as a rough example) 99% confident at the macro level, 98% confident one segment down, 97% confident two segments down, 95% confident three segments down, etc.

One way to handle this is to build the test from subsegments up to the macro level.  Let’s say at a minimum you want 3 subsegments in the test – retention, lifecycle or tactical / seasonal – and each of these you want to be 95% confident in.  Since some of these programs are triggered by behavior (lifecycle) and some by calendar (seasonal) I’d guess the sizes of the populations and number of executions could be vastly different.  Meaning, you may only need 1% control on the seasonal promotions but more like 5% or 10% control for some of the lifecycle stuff to be 95% confident on the outcomes of those.

When you sum all these segments up, you often end up with more like 2% or 3% of the entire population in control groups to always be at least 95% confident at all the desired subsegments, which means you end up with even higher confidence at the macro “all campaigns” level – a very good thing.

And much better than trying to explain why you can’t answer a subsegment question because you used 250K instead of 400K or 600K in the control group, if you know what I mean!  That’s when people forget the arguments about foregone revenue and start saying stuff like “Why did you not use a larger control group for this test?”

In the end, you will thank yourself again and again for using a larger than minimum required control at the macro level because you WILL come up with that unexpected “must know” question and be thrilled to find out you actually can answer it at a decent level of confidence.

Good luck with it, let me know what you learn!

Jim

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Choosing the Size of Control Groups