Has to be There

I find it really interesting that whenever there is a discussion of measuring the value of social media, there’s such a bias towards believing there is value in social beyond what can be properly measured.  See the comments following this post by Avinash for a good example.  Speculation is fine, but the confidence being expressed that a new tool or method will uncover a treasure trove of social media value seems un-scientific (as in scientific method) at best.

I don’t doubt there is some value in social media beyond what can be measured, as this has been the case for all marketing since marketing measurement began.  These measurement problems are not new to social either:  Marketing value created is often situational, it depends on the business model and environment.  What works in one situation may not work in another.

For example:

To small local businesses who do no other form of advertising, there is a huge amount of relative value to using social media, versus no advertising at all.  Social advertising is much better than none, and since it’s free, the incremental value created by (properly) using social is huge.  It’s also really easy to measure the impact and true value, since the baseline control is “no advertising”.  Lift, or actual net marketing performance, can be pretty obvious in his case.

On the other hand, many companies are running lots of advertising designed to create awareness, and the incremental value of social as a “media” may be close to zero for these companies, or at least less than the cost to analyze the true value of it.  Possible explanation:  Social events such as “Likes” or comments are simply representations or affirmations of awareness already created by other media, so by themselves, create little value.  In other words, events such as Likes might track the value of other media spending, but may not create much additional marketing value.

Why is this plausible?  It seems unlikely a person would “Like” a brand or product if they have not already experienced it, and are already a fan.  This means in the vast majority of cases, little incremental awareness / acquisition is created.  If this case is not true, if people “Like” a company even though they have no reason to (paid to Like?), then the problems with social marketing analysis go way beyond tools – the concept and data driving the analysis itself is flawed.

But if Like really means Like, the number of Likes or any other similar social events do not have as much to do with awareness as they do with the size of a loyal customer base, and are much more aligned with tracking the success of other awareness / acquisition campaigns.

Looking for Love in All the Wrong Places?

That all said, I believe there is some value being created in the acquisition / awareness area from social.  The problem seems to be this value, when measured, is quite a bit less than everyone expects.  So “the hunt for social value” seems never ending, with speculation and measurements contrived from thin air immensely  popular.  This missing value just has to be there, right?

The core problem is an old one: online value measurement definitions are all over the map, so it’s easy to claim value was created by simply inventing a new way to measure success.  I can’t wait for the day when established test and measurement standards (like using control groups) are adopted in the online space.

Meanwhile, I think it’s quite possible if people keep looking in the awareness / acquisition area, the value of social “beyond what can now be measured”, in many cases, is probably less than the cost of actually measuring it.

Why this approach?  Based on my experience, People are Social, Media are not.  So if you want to derive social value, you use people metrics, not media metrics.

Using this approach, I have unbridled optimism for the value of social.

But I won’t go as far as insisting value is there without measuring it properly first.  Because that’s not how science works.

See ya at eMetrics NYC!

P.S.  There’s lots of real experimental science out there on the effects of social media in the marketing space, have you reviewed it?

You will find this material to be a treasure trove of new ideas and proper methods worth pursuing in the social measurement space, examples here, here, here, here.  Get yourself a subscription to Marketing Science or if you are a WAA member, you can request copies of these fully documented social measurement experiments.

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

“Missing” Social Media Value

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: We’ve been playing around with Recency / Frequency scoring in our customer email campaigns as described in your book.  To start, we’re targeting best customers who have stopped interacting with us.  I have just completed a piece of analysis that shows after one of these targeted emails:

1. Purchasers increased 22.9%
2. Transactions increased 69%
3. Revenue increased 71%

A: There you go!

Q: My concern is that what I am seeing is merely a seasonal effect – our revenue peaks in July and August.  So what I should have done is use a control group as you described in the book – which is what I am doing for the October Email.

A: Yep, that’s exactly what control groups are for – to strain out the noise of seasonality, other promotions, etc.  But don’t beat yourself up over it, nothing wrong with poking around and trying to figure out where the levers are first.

Q: Two questions:

1.  What statistical test do I use to demonstrate that the observed changes are not down to chance

2.  How big should my control group be – typically our cohort is 500-800 individuals

A: Good questions…

On a group that small, you are probably not going to get anything “statistically significant” without ruining your total profit, e.g. might have to use 50% in control.   If you have the leeway to do it, that’s what I would do.

On the other hand, in some cultures people will go bonkers over giving up sales to learn something really important.  OK, so take 10% as control and repeat it 3 times; if the results are stable then you have your proof.  Do another control every once and a while (every 6 months?) just to make sure it tracks.

Either way, you don’t really need statistics.

Practically, confidence is the likelihood a sample represents the population.  This can be a really useful idea when you are forced into very small sample sizes or the event is highly risky to repeat.  But here, if you are testing a really large slug of the population, confidence is less useful.  Or if you can repeat the event (because essentially, you are in control of it and it’s low risk), do you really need to force yourself through the screw of complying with the statistical math?  It’s like using a 727 for crop dusting, overkill for the situation, methinks.

If you were running a drug manufacturing line, statistical concepts like confidence and significance are absolutely valuable.  But for a marketing program?

That’s why I love the idea of “beefy controls” in start-up projects because I *do not* have to rely on statistics that the audience likely does not understand and provide room to question the results, e.g. “Yea, but what if the result is an outlier?”  Very appropriate in high risk situations, with giant populations and a lot of money on the line.  For this situation, perhaps not.  But, if you’d like to go that way, there’s lots of calculators on the web that let you play with some of the numbers anyway.

Here’s one, make sure to read the descriptions of the variables underneath the calculator:

http://www.surveysystem.com/sscalc.htm

Nice work on the core campaign idea, by the way!  Now we just have to tighten it up a bit…

(3 months later)

Q: We decided to tighten the targets and do a “best customer defection” email program. Basically, we look at customers who has an RFM score of 555 in the previous scoring period who have dropped out of that score.

A: Interesting!  So instead of targeting by  guessing the current score of a defecting best customer (say 355), you are looking for all customers who were formally best customers, regardless of current score.  This is a subtle difference, but much more of a LifeCycle approach and frankly why I prefer these kinds of ideas over “straight” RFM.

An example might be helpful.  Let’s say the acquisition folks run a huge new customer campaign in between the prior RFM scoring and the scoring done before your campaign drop.  A big inflow of new customers can artificially “force” certain groups of customers down in score – even though their own behavior has *not changed*.  In this case, the new score is not reflective of actual behavior, so increases noise in the system.

That’s the problem with the “Snapshot” or date-specific view of Customer State – it’s a single point without reference.  By using prior score, you are acknowledging behavior over time and the primary importance of the former State, as opposed to the current State – a Movie as opposed to a Snapshot.

In other words, from a  Marketing perspective, I’m more interested in the path they are taking through the LifeCycle than any particular point in time during the LifeCycle represented by a single RFM score.

Q: Good news on your advice.  We ran a 50% control (500 purchasers in each group) and the results really nailed the issue for us. The actual number of purchasers remained unchanged at 20% but Total Revenue and Average Spend increased by 40% compared to control.

(Jim’s Note: for those not following, a very precise target group of 1000 was split into 2 groups of 500.  One group received this  campaign, the other did not.  People who did not receive the campaign purchased at the same rate as people who did receive the campaign, but the people who received the campaign averaged 40% higher spend).

A: Awesome.  So what you are seeing is Customer State makes a huge difference in terms of what offers / timing can be most effective for this “Recently defecting best customers” cohort.  If I’m reading your numbers correctly, no lift in response versus control but a huge lift in revenue.

To me, that means these customers are early in the process of defection – still buying, but without a special treatment, slowing down the monthly spend.  After all, they are very Recent (former 5XX), so highly likely to purchase again, which is why lift in response was flat – they likely would have purchased anyway.

Not a bad time to hit them.  Offers to a very Recent State should focus on increasing order value, not generating response – you don’t want to spit into the wind, but go with the natural flow of the behavior.

In other words, these customers likely would have purchased anyway, but at lower price points if they had not received the campaign.  The common way this is addressed is with  “threshold” discounts – if average order is $50, then something like “$10 off any purchase over $50″ – test different thresholds to maximize profitability.

Looks like you gave them the right offer ;)

On the other hand, a straight discount to this specific best customer group – $10 off anything, and especially when their normal category of purchase is promoted to them – almost ensures that you will lose money.  Why?  Most of these customers would have bought at full price anyway, as demonstrated by equal buying activity whether the customer received the campaign or not.  So the discount turns into a loss versus no campaign at all.

Unfortunately, I see a lot of this exact type of campaign delivered to best customers because all customers get some version of the same offer.  “Hey Jim, we’re not sending the same message to every customer, we send different messages by segment”.  Sure, the copy and art are customized for different segments, but the segmentation is not by Customer State, so the offers are mismatched and suboptimal.

This is the value of using control groups; they drive understanding of Marketing concepts like opportunity costs and subsidy costs.  These two concepts are the reasons why ignoring Customer State is suboptimal: by not segmenting using State, you will get lower than possible profit or sales at most customers, depending on Customer State.

Had you not delivered a campaign tailored for prior Customer State, money would have been left on the table by way of lower order size.  And 40% Revenue lift sounds like it might have covered the cost of the campaign ;)

Q: We tried to run a Student T test on the results but our new statistician informed me that the distributions were not normal – so on her advice we ran a Wilcoxan Test which gave us a highly significantly p = 0.016

A: Oh, so you still went the stats route?  Well, the fact you HAVE a statistician tells me the culture there is more familiar with interpreting these ideas, so more power to you.

Glad it worked out and keep me informed on how things go downstream.

Jim

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Increase Profit Using Customer State

Before every eMetrics (I’ll be in San Fran teaching Basecamp, at the Gala, etc.), I try to ask myself, what is the most critical issue facing the web analyst community right now?  Then, at the show, I ask everyone I run into what they think about this issue.

There’s lots of issues to choose from.  Career path I think is a big area of discussion, given the mergers in the space and trend towards outsourcing.  Then there’s the “we don’t get no respect” thing; senior management doesn’t seem to listen / understand / act on the information provided.  And one of my favorites from the past is still out there, data torture - people being pressured to manipulate data to reach a predetermined analytical outcome.

But seems to me, more important at this juncture is trying to resolve why there is so much written about the importance of “the customer” but very little measurement at the customer level.  Think about it.  Customer experience, customer centricity, the entire social thing, it’s all about customers.

But when folks wants to trot out “proof” that this or that approach is the road to the promised land, they analyze impressions, visits, clicks, etc.  Visitor-level stuff.  Does that seem like the correct approach to you?  Seems to me, if you want to provide knowledge about customers, you should measure customers.

One thing we know is customers do express behaviors through a web interface that are not relevant to the future behavior and value of the customer.  One of the earliest and most widely publicized incidents of this was with Amazon gift purchases.  People went on and on about buying a gift from Amazon unrelated to their interests yet having that category Marketed to them relentlessly over time, even though they never purchased from the category again.

This problem was eventually solved by Amazon using Recency, a classic customer behavior metric – only more Recent behavior was used to make suggestions.  Recency is predictive; and lack of behavior is often just as important, if not  more important, than expressed behavior when trying to understand customers.  Unfortunately, most web analysts are trained to look for expressed behavior, not the lack of behavior.

Further, just because an event of some kind happens in the stream of web activity does not mean the event had any affect on the behavior of the customer.  Display impressions, searches, social interactions, all of it – how can you tell whether the event had any effect on the customer at all?  The only way is to measure at the customer level, for example, comparing the behavior of customers who were exposed to the events with customers who were not exposed.  Or, modeling different mixes of events against customer behavior over time, a “marketing mix” model of sorts, to stretch the idea.

Now some people are going to say. “But Jim, we don’t have web tool access to this data!” or “We don’t pass web data to the back end” and all manner of other related excuses, to which I would say,

“Where is your curiosity?”

Clearly, a unified database is best.  But just because your company can’t afford an advanced WA tool doesn’t mean you can’t do this.

I mean seriously, get a dump from the order management system into a spreadsheet.  Run a query against the CRM database.  Look up individual cases in the customer service or lead management systems.  This the way analysts make breakthroughs, how  business cases are built.  If key web data (campaign codes, logins, etc.) doesn’t make it into the back end, why?  If form data crosses over, how hard could it be to send a campaign code, login, or other critical data?  With proof, then pitch the advanced WA tool, or systems, processes, people, whatever you need to make it easier to analyze customer level data.

OK, so let’s hear all the reasons why it’s fine to draw customer-level conclusions using visit-level data, or why you can’t do the above, which I’m sure will include some of the following:

1.  My boss doesn’t care about customer-level data, ignorance is bliss, pseudo-analysis is OK

2.  I’m too busy learning very little about a lot of things instead of going deep on the most important stuff

3.  Shiny objects rule, so see #2 above

4.  I’m a web analyst, back-end data is not my thing

Other reasons?  What do you think?

Do you see the hypocrisy in claiming to understand customer behavior based on visit behavior?

Let’s talk about this at eMetrics San Fran…and Toronto too.

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

All Talk, No #Measure

Just because you pay $5 / CPM for impressions, does not mean they are worth $5 / CPM, does it?  Do people really still have that kind of mentality?  Is the price of the media equivalent to its value?

For example, I’m sure you have heard of multi-million dollar campaigns that generate very little lift in sales.  Happens frequently in fast food, for example.  What is the value of that media?  Is it the millions paid?

What really blows my mind about this approach is it’s so offline, so old school PR. Do the folks who put forth these kinds of models believe nothing has changed in 50 years?  What happened to the whole rap of online being “different”, that you can’t measure it like offline, blah blah.

Except when it’s convenient to do so?

If you want to know the value of a Facebook fan, why not measure the value of a Facebook fan?  Because it’s hard, and would require organizational discipline?  Too bad.   Substituting the kind of models used in the example above for actually measuring the value of a Facebook fan is misleading at the very best.

Make sense?  If you’re with me on this line of thought, let’s not stop here.  We should go ahead and question the value of awareness.

Now comes a better view, but likewise,  just because an event happens does not mean it has value or contributes value.  Looking at the recent post Social media and SEO massively undervalued: study we see a great data collection effort through TagMan but a similar premature jump as above:  that because an event occurs, it somehow must contribute value to the final outcome.  Again, this is a very old-school idea being applied to an environment where there really is no need to guess; set up a test and measure it.

I realize people get excited by the potential of new applications and tools, but have to wonder why folks are so willing to throw logic out the window and “find an answer” even if they have to torture the data to do so.  In many cases the reason is promotional, to sell a product or service, and hopefully this is pretty transparent to the reader.

One of the big problems at the root of all this is the lack of a common value reference point.  In other words, a standard that can be applied to compare the relative value of impressions, events, touches, opens, clicks, and so forth.

This standard exists, it’s called a controlled test.  In academic environments, where all the studies, results, and conclusions are peer-reviewed before they are published, it’s the gold standard for determining “the value of”.  This is a particularly important concept when you are dealing with interactivity; the results of controlled tests can be surprisingly different from common perceptions.

Perhaps it’s time for this community to require (OK, at least ask for?) the same level of transparency.  Count me in.

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

But What is an Impression Worth?

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: Thank you for creating this useful website!

A: You’re welcome!

Q: When figuring out retention rate for an annual or a 8 months life time cycle period, how do I pick the starting period?  Do I look at their first orders on a date?  Or I pick a time frame such as one month?

A: It depends on:

1. What kind of “retention” you are talking about, the definition, which is probably impacted by the audience for the data

2.  What you will do with the retention data, what kind of decisions will be made and actions be taken because of the data

You should always ask these questions above  when someone requests “retention data” – or any other kind of analysis, for that matter!  For example, there probably is a huge difference in what you would provide to the Board of Directors for an annual benchmark and what you would provide to Marketing people for executing campaigns.

In the first case, the data would probably be used to inform Strategic decision making, for example, should we change our product mix or approach to pricing given the market?  In the second case, the data would probably be used in a Tactical way, for example, to target new customers who are predicted to defect because of the campaign they responded to or the product they bought.

If providing data to the Board, “annual retention rate” would probably make the most sense (again, you should ask, what’s it for?). If that’s what they want, you would pick a starting period, probably aligned with the fiscal year (Jan – Dec?), and find out what percent of people who purchased Jan – Dec 2009 also purchased Jan – Dec 2010.

That’s the annual retention rate.  Useful information, perhaps leading to the Board requesting action of some kind.  But by itself, you really can’t “do” anything with this data, there’s no source or targeting information, there’s no customer value information.

However, if you segment by campaigns, product of initial purchase, price points, offers, or other actionable variables, the retention rate could be just about any formula, e.g. what is the retention rate:

a. Today, of people who made their first purchase in 2005?
b. End of 2009, of people who made their first purchase in 2005?
c. Today, of people who ever bought Product X as their first purchase?
d. Today, of people who bought Product X as their first purchase in 2009?
e. Today, of people who had at least 2 service calls in 2010, who became new customers in 2009, who used a 50% off promotion?

and so on.  Retention rate for anything tactical almost always requires and audience and time frame to be defined.

Q: You mention in your article, “Total number of customers” as the denominator for calculating the customer retention rate, do you mean the total customers at the end of the period?  Or those total customers came in on the first date of a fixed period?  Or the first fixed period that I’m observing?

A: Whatever definition is the correct definition depending on the need of the audience.  There is no standard, other than perhaps the very first one, the Strategic “reporting” idea of year over year retention.  This is commonly used in reporting to Wall Street, for example.

While discussing this particular idea of “customers”, one might encounter the common problem of not knowing the definition of a customer, at least in terms of retention.

When does the company declare a customer is no longer a customer?  Is a customer  “everyone” who has ever purchased?  If the company has been around 10 years, and you are calculating retention rate “today”, as in how many of these total customers purchased in the last year, you may find you have a very low number, one that won’t mean much to anybody, and is not actionable.

On the other hand, if your definition of “customer” includes a level of activity, for example, “must purchase at least twice, one of those purchases in the past 3 years”, now you are talking about a highly actionable kind of retention definition.  Why?

Because there is some hope that people who have purchased at least 2x (Frequency), at least once in the past 3 years (Recency) could actually still be customers, as opposed to defected customers.  If I am calculating a “serious” retention rate, something to be used to take Marketing action, or pay out bonuses, or revise policies, I want to measure against people who actually have some Potential Value, some Value to the company in the future.  That’s how I define a customer.  To me, there isn’t any point in calling someone a customer who is unlikely to contribute to profits in the future.

If you define as a customer “anyone who purchased over the past 10 years”, you just have a dead metric that really does not reflect the reality of what taking action might produce.  In other words, you are including people who are extremely unlikely to still be customers, so what’s the point of the “customer retention metric” you created?

Does the above help answer your question?

Q: I wasn’t expecting you to reply me so fast and in such detail!!!  Thank you so much!  I’m calculating this retention rate for marketing and your answer is very helpful for me!!!

A: Great!  So maybe ask them specifically how they want to look at it, and if they seem puzzled, suggest to them various options.

I can tell you from experience with businesses like yours is the buying behavior tends to peak early and you have to act quickly if you want to extend the lifecycle.  Perhaps not quite as time-critical given your “triple bottom line”, but probably not too different.

This argues for a tighter leash on the definition of a customer, perhaps purchased at least twice, one of those past 6 months.  You could also do 2x purchase, at least once in past 3 years, and compare, it will give them a feel for customer defection trend / rate.

The next step would be the Lifecycle map, which uses Recency and Frequency in a more actionable way, like this example.

Marketing people should be able to use this map to target specific groups of customers, e.g. purchased 4 – 9 times, but not in the past 90 days.  These are good customers who are in the process of defecting, and require special attention to keep them on board.

After all, the point of measuring retention is not retention rate itself, it’s about increasing the productivity and profitability of the business system.  Just as you can optimize for conversion, you can optimize for retention, and sometimes you discover they conflict.

For example, one company I worked with featured certain products on their home page because those products had a high conversion rate on visits to the home page; they had “optimized” the home page for this scenario.

However, a very quick and simple calculation showed these products generated customers  with terrible repeat purchase rates relative to just about every other product with volume.   A quick survey of these customers found out why the repeat purchase rates were so low – almost all customers disliked the product and thought the company deceived them.  Turns out the company “over-sold” the product – and that’s why the high conversion rates.

In another case, PPC campaigns had been optimized for conversion without regard to customer retention.  Under a budget crunch, the lowest converting campaigns were killed, but overall sales volume over the next 3 months dropped much more than the sales generated by these campaigns.

Reason?  These low converting campaigns generated the company’s very best customers in terms of 30-day, 90-day, 180-day value, while most of the highest converting campaigns generated low value, single purchase customers on the same time frames.

This kind of analysis is simply not that difficult to set up and execute, relative to the extreme amounts of value that can be created:

1.  Pass campaign codes / info with the order to the backend order processing.  If you are not doing this yet, start right now!

2.  Select a campaign, choose a time frame.  If you want to match up to financial statements (a good idea if you will be talking to C-Level folks), say January 2010.

3.  Grab all new customers who came in on Campaign X during Month Y – what is their average value 1, 3, 6, 12 months later?  This is a Lifecycle by Campaign analysis, similar to the LifeCycle map example mentioned above.

The new customer experience (channel, offer, product) is one of the most powerful predictors of future customer value, and the value of these new customers relative to each other tends to remain stable regardless of how many other generic campaigns (weekly email) you throw at the customer over time.

Across all campaigns, about 60 – 80% of these new customers will have the same value at 12 months they had at 1 month.  The question to answer, as with any optimization, is this: knowing the customer value created by these campaigns varies widely, are we allocating the acquisition spend optimally?  For example, are we spending 70% of the budget to generate  20% of the annual customer value?  Are we willing to pay more for clicks that generate new customers with 10X higher annual value?

Retention rate isn’t just some mystical number, retention rate quickly turns into profit dollars and can have incredible financial impact!

Jim

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Optimizing for Customer Value

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: First off, I very much appreciate you sharing all this wonderful content on your blog and conferences such as eMetrics.

A: Thanks for that!

Q: My question is a simple one, but I think the answer may be hard: When does a visitor “need” a coupon?  *Need* defined as: visitor would not have placed an order unless presented with the coupon.

A: Hmmm…methinks we’re going to have to define a few concepts and be clear on the goals to make sure we are nailing this down… visitor versus customer, sales versus profit, etc.  In other words, answer is not hard, but could be complex without defining context.

Q: It’s still a mystery to me why so many retailers seem more than willing to hand over all their margins to Groupon or give coupons to basically all visitors.  I am curious whether you would approach this question using  observational data (eg web analytics) or experiments (eg AB testing), or both.

A: Right – is a mystery to me too!

There are certain situations where this approach might be appropriate, but the problem with much web “marketing” (which often is really just advertising without much thought about marketing) is often there is success in a narrow or special situation.  Then the pundits jump on and say “if you’re not doing this you are stupid”, regardless of the business situation and / or without recognizing the special circumstances that are driving success.  This is all the real Marketing stuff people leave out; understanding why it works, under what circumstances, for which segments, involving which products.

When you don’t really understand what is happening and why, no learning takes place.  When no learning takes place, everything is a “new idea” and people are surprised when the outcome is different.

So, for example, if I was launching a brand new service business (restaurant) or a new product that is complex and won’t sell without trial (like yogurt that “naturally regulates your digestive system”), Groupon might be a slam dunk for a product launch.  No surprise here; coupons are often used to drive trial in product launches because the need is to reduce price resistance and drive sampling.

On the other hand, if my product or company is well known and I have tons of loyal customers, Groupon could generally be a financial disaster if you care about profits.  But if all you care about is response to the coupon, it could be a great success!  Because tons of people who would have bought at full price anyway get a huge discount and you get to sell the product below cost.  Awesome!

What do I mean by this, how can you have high response and low or negative profits?

Here is what I have seen over the years: whenever response rates are abnormally high, it means you have a high percentage of responders who would have bought anyway without the coupon.  This is seen over and over in database marketing, online and offline.  From a financial perspective, it means you have probably given up the coupon value with no benefit, a so called “subsidy cost”.

How do you prove this is happening in a promotion?  If you want to really look into and prove these effects, first examine the percentage of response that is from current customers.  If it’s high, that’s the first clue the discount is cannibalistic, not incremental.

If you want to quantify this subsidy cost a bit more is a relatively simple way, take the customer redeemers as a group and look at their average sales for a few months prior to the coupon promotion, during the promotion, and a few months after the promotion.  Often what  you will see is their spending behavior changed very little during the promotion.

For example, let’s say the coupon is 50% off.  The monthly net spending sequence over time might look similar to this:

2 months prior: $100
1 month prior: $100
Promotion month: $50
1 month after: $100
2 months after: $100

This shows the customers redeeming the 50% off coupon did not change their behavior at all; they simply took the discount and bought what they would buy anyway.  Meaning, the coupon cost is a real cost to the bottom line with no offsetting incremental profit.

Bottom line, for every response you lost $50 in sales plus the cost of the campaign, even though you had tons of responses and sales from those responses.  If it was a large campaign, your overall sales for the month net of discounts probably dropped.  Financially, if your cost of goods is 50%, you gave up $25 in profit for every response, minus the per response cost of the campaign.

And, the above behavior is most likely to occur with best, most active customers!  Across all redeemers, you might get $60 or so instead of $50 during the promotion month, but you are still losing money on every redemption – the higher the response, the more money is lost!

This is one big difference between Advertising and Marketing.  Marketing goes beyond Advertising, wants to understand the relationship of specific products to segments of customers, how pricing and modes of distribution affect this relationship, and the profitability of the relationship.

So, with that backdrop, let’s try the question:

Q: When does a visitor “need” a coupon?

A: If I take your question literally, there is a concept in Marketing called coupon proneness, and it’s the classic definition of “needs a coupon”.  Essentially, it means the more coupons you give people the less likely they are to buy without one. If you can imagine what this looks like over time, it’s margin erosion hell.  It’s taking the example above, where no incremental profits were generated, and ensuring it will happen time and time again.

From a Brand perspective, always offering coupons means you are teaching people your prices are too high, or there is a tangible reason your company has to “beg” for sales (implies poor service or quality).  Either way, the outcome is not so good for Brand trust and any evangelism that might result.

The exception to the above is among the “never pay full price” segment, who don’t buy anything without a discount / coupon.  From this segment, you get the benefit (?) of your coupon offers being spread all over the web, attracting many other “never pay full price” customers who generally have negative net values to the company.  Great, huh?

The end result of this pattern is horrible customer loyalty, margin erosion among current customers, and lots of new customers that are 1x buyers.  This means you have to spend *even more* on Advertising to constantly chase new 1x customers, while at the same time your margins in the current customer base are being consistently eroded.

Certainly not an optimized system!

Some people will argue the “extra sales” they get are worth the price of encouraging the above behavior.  But there are not too many businesses that put sales in the bank, what they put in the bank are profits.  So this is very short-term thinking and in fact, you find a lot of businesses that follow this model perform very poorly financially.

So, you ask, when would a visitor “not have placed an order unless presented with the coupon”?  The answer is this: when you have “presented” a coupon before, and the more often you have done this, the less likely they are to buy *until* you present one.

Sure, you could use A/B testing, but it’s not hard to guess what you will find – when you present a coupon, more people buy.  Duh, that’s Advertising, right?  But that’s optimizing for conversion, not for profits, and conversion can’t be deposited in the bank any more than Sales can be.  You have to go further.

For example, if you had the capability to recognize purchase or visit patterns among visitors, you could segment by these behaviors and present coupons on the site only when they were likely to have an incremental rather than cannibalistic outcome.  For example:

1. A new visitor who becomes a repeat visitor X times but does not buy

2. A current customer who has not purchased in over X weeks

and so forth.  You could test for “X” and optimize for highest profitability if you also ran a “null” control group  – where if A = coupon, B = no coupon.  Then look for incremental sales  behavior or “lift” from those offered the coupon versus those not offered a coupon, and run out the profit and loss.

Of course there are other scenarios, mainly current customer comes to your site because  you sent them a coupon, as opposed to presented one on the site.  Not sure if you were including that in your question, but I took your meaning literally.

The scenario with subsidy costs when sending customers a coupon is basically the same as the example above, except you control which customers get what coupon values or if they get a coupon at all.  More info on executing and measuring in that scenario for customers here, and for an example of a company putting this approach into practice, see here.  You can really drive higher profits by doing this correctly.

Jim

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

When Does a Visitor Need a Coupon?

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: I was wondering if you’ve done any work with, or given thought to, companies who have a cloud based Freemium business model?

Should they be tracking usage (or anything) at the free level?  Should they be tracking usage at the paid level?  I’m sure defection rates are a big problem, but I’m wondering how many focus on engagement thru mass marketing versus trying to keep what they’ve got, or influence the free users to make the leap to paid.  Any thoughts on this?  Maybe you could do a blog post on it.  It seems like a good fit with your brand of analysis but I’m just starting to think it through…

A: I just finished an analysis that’s a good example of this problem.  Behavior during the Freemium period can predict who is highly likely to become a paying customer, who will need marketing efforts like additional sampling / package discounts, and who will not become a customer no matter what you do.

So the answer is you need both, analysis of paid and free.  But in particular, what you need to do is understand the transition from free to paid by comparing the behavior of known converters versus non-converters over time, preferably using events that create value for customers.

Typically the differences will be volume / persistence related, generically, low Recency high Frequency.  Also, likely converters to paid will tend to use a wider variety of features consistently.  In the analysis, the question to answer is which of these value-creating behaviors is predictive of becoming paid?

Said another way, you tend to see a fairly fast drop-off among *all* new Freemium customers after the initial burst of activity, but the ones that are not going to stick tend to drop off even faster in their activity.  Then, there is a “bounce” in activity where the ones who are most likely to end up as paid begin to cycle behavior more quickly and begin to use more features, and the others simply drift off the map, with no “bounce” as Recency becomes extended.

Classic LifeCycle analysis – customers tell you what they will become in the future by what they do today.  Having the very detailed behavioral information typically seen with interactivity just multiplies the ability to do this kind of prediction.  More on the Freemium model, including determining appropriate cost to acquire, is here.

Q: Do the standard analytics packages allow a business to look back at the “free” behavior of paid subscribers?  I’m thinking of Freemium cloud based solutions and how they would track this.  Do products like Crazy Egg get you there or do you really need something more sophisticated to do this kind of analysis?

A: I’ve never used Crazy Egg so I don’t know about that one specifically, but in general you can do quite a lot with the basic tools that support customizable segmentation.  The challenge with going that way is you have to be super-technical with the implementation to capture important event data points, you have to create many different segments, and then the killer problem – you can’t “re-analyze” a different approach with these tools, for the most part.  If that’s what you mean by “look back”, it’s highly unlikely you could accomplish what you need to do.

So it’s possible, but these tools are not really designed for “behavior over time” work and certainly don’t allow for much “exploration” of the data – any change in analytical approach is likely to be a “going forward” type of measurement, not looking back.  So there would be lots of iteration before you even knew if the analytics set-up was correct or what events are meaningful.  In other words, it’s possible but could waste a LOT of time.

I’d much rather find the system that contains the key elements of activity – when did they sign up, what features are they signed up for, when did they add other features.  This data probably resides in whatever system manages the account.  Dump that data off into a spreadsheet or database, try to figure out what’s meaningful, look for correlations.

Then, once you have a grip on some solid ideas, then maybe you go into the front end and try to align traffic and behavior with known “events” that seem to predict upgrade to pay, if that’s what the mission is.

Otherwise, you will be setting up a ton of tracking on all kinds of events not knowing what is meaningful, and then dealing with a really poor interface for the analysis of those events.

The other way to go, of course, is to use one of the advanced web analytics tools, which sit on real databases and can be queried.  But assuming that’s not an option, I would try to look for hard data points in the backend first, then knowing key behaviors, look for what might cause those behaviors in the traffic side.

Example

Let’s say you have a project management application that has a 60-day free trial then converts to paid.  Value is created for the customer when they use the functionality of the app – say create project, comment on project, upload file, or any other actions you deem critical.  “Traffic” in a situation like this may be only marginally indicative of value creation; rising activity could even be a negative indicator (frustration with application).

So, you want to create a situation where you can analyze these important behavioral events by account, and (ideally) you want to know the source of the account creation – campaign code, referrer, etc. That’s all you need for data, simple table, maybe a dozen columns.

Then, compare average account that converts to paid with average account opened at the same time as the converters but does not convert, over the 60-days before trial end.  For each of converting and non-converting, aggregate each of key events by week, divide by number of accounts to get average behavior per account, and you would have 8 weekly average data points for each of the events, both for non-converting and the converting accounts.  Maybe a dozen simple line graphs with 8 weekly data points, one set for accounts that paid, one set for accounts that did not.

Converting and non-converting graphs should look different for some variables.  Both will typically start out with high levels of activity, then for some variables you will see them diverge.  This not only predicts which variables affect conversion, but reveals to you the best time during the 60-days to intervene with surveys, help, or other marketing programs to re-engage the accounts that appear to be headed for defection.  If you have campaign data, also which campaigns tend to create accounts that convert and which don’t.

One of the event graphs may look to be more predictive than the others, with abrupt changes in direction going into the conversion event.  For example, perhaps it will look like this:

(Click pic for larger image)

This is the behavior of 10 different 1st year spend levels (deciles) over the first 14 weeks of their Life, engaging in an event that creates value for them.  The dark blue line represents average top spender.  Note how for top spenders, the profile is quite different.  The graph tells you that by week 4 or so, you can probably predict who will become a best customer and who will need intervention based on this activity.

You can run this kind of event profile for any variable – events, campaigns, etc. as long as you know complete / non-complete goal or end value of the customer.  In your case, since the goal outcome is binary, there would be 2 lines instead of the 10 spending deciles above: converters versus non-converters.  Create a converter versus non-converter chart for each key activity variable (create project, comment on project, upload file, or any other actions you deem critical), and look for this kind of divergence.

Drilling down more deeply by excluding all but 3 lines so we can see the behavior “in the middle”, we find some interesting patterns:

(Click pic for larger image)

Here, we see patterns that provide clues to the testing targets one might want to address to see if “middle” customers could be turned into better customers.  The blue segment, showing a series of higher highs and higher lows after it “bottoms” for this behavior, is most likely to benefit from intervention of some kind.  The pink segment looked promising, but then put in lower highs and lower lows – these customers lose momentum quickly and have trouble self-sustaining.  The yellow segment was never really in the game at all.

Yes, the comparison to stock market charting is intentional!  It’s an expression of group behavior.

If I had to pick the segment with the best potential, I would try the blue segment first, and the data points could be used for automated triggering of different types of campaigns. For example, “If by week 4 activity for Variable X  falls below 60, trigger Campaign A.  Then if by week 11 activity for Variable X is not above 40, trigger Campaign B.”  Remember, these are averages, so not all customers in the segment are below threshold.  The idea is to target a specific behavior with a specific message.

Just to be clear, you don’t need the goal value of the customer to put a model into practice, only to prove the initial model – certain patterns in behavior predict high value customers.  Once you know the end value of  customers – convert or not, monetary value, any goal – you can run the LifeCycle movie “backwards” like the charts above and find out which early  behaviors are predictive of high and low value customers.

If you want to go further, you could show these graphs and data to a modeler and see if they can create a more precise mathematical model, which can be developed much more quickly with this kind of evidence to review.

Once you fully understand what this LifeCycle landscape looks like, THEN you could go back and instrument the web site and analytical tool to monitor some version of this data in a more automated way.  But trying to guess what’s going to be important beforehand and work through a study like this using a vanilla web analytics tool is the very, very long way to get where you need to go!

Jim

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Freemium Customer Conversion

So here, I will attempt to answer other kinds of questions I think people might have based on my discussions with WAA members.

Update: The WAA has answered many Certification questions here.

However, I’m going to approach this topic a bit differently than most of the published documentation – from a Product / Marketing perspective, rather than an Educational / WAA POV.  I can do this because (if you don’t know) I have worn all the hats on this project – developer, marketer, WAA project owner – and I think it might be helpful to tell the business story of the WAA Certification, from the bottom up.

And if you have other questions, feel free to leave them in Comments and I will do my best to answer them!

Where did the idea for Certification come from?

The WAA is a member-driven organization; we listen to the membership and try to accomplish what they would like us to accomplish.  We heard from hiring folks and managers that ”web analysts today know a lot of the buzz words and can follow instructions as far as reporting goes, but what we’d be willing to pay a premium for is web analysts who discover things on their own, who add value in areas we don’t already know about”.

So that’s where WAA Certification came from.  It addresses a specific need identified by members, what came to be known internally as the “Book Smart versus Sherlock Holmes” problem.  Sure, you can read a ton of books or blogs and be a  good web analyst by following best practices.  But so can a lot of other people.  What you need to pass the Certification Test is different; you have to be able to turn data into insight and recommend a best action given the scenario presented.

This is Why the WAA’s Educational efforts lack “tool focus”?

Sure.  And of course, the tool vendors already own that focus, and by definition they have the resources to be much better at tool education / certification than the WAA, so why would the WAA want to compete with the tool vendors in the same space?

Better to add value on the business side, where there is demand we can fill and a lack of trusted resources.  And if you think about it, this approach simply expands the overall WA opportunity.  People who want to become experts on the tool side have a path (through the vendors), and people who want to become experts on the analysis / business side also have a path through the WAA.  And if you want to be a Universal Web Analytics Soldier, I guess you could do both!

Does that mean I can pass the Test with No Tool Knowledge?

Not at all.  The threshold we set is you need to be able to communicate effectively with tool experts to pass the test.  That means you will need to know the basics of how the web works, how the tools accomplish their mission, and know what all the web analytics terms mean.  Example: To pass the Test, you don’t need to know how to write a tag, but you do need to know when a  custom tag  is required and how to communicate your need effectively.

So Marketing people can Pass the Certification Test?  eCommerce Managers?  Usability people?  Media Buyers?  Etc.?

Absolutely, if they are good at transforming the data generated by web analytics tools into business insight AND have broad knowledge across the entire scope of web analytics.

Who Created the Certification Test and How?

About 50 WAA members from all over the world volunteered to take on the task.  We created questions, tested them across different audiences, gathered feedback, rewrote the questions based on the feedback, tested the questions again.  You know, the continuous improvement thing?

If you want to participate in the ongoing process of creating the Certification Exam, there is more info here.  Please note you have to be a member of the WAA to be on any WAA Committee.

Where did the Requirements to take the Test Come From?

From the 4 Test the Test sessions we held at various eMetrics events, where we asked people to volunteer to take the Test.  We looked at the backgrounds of  people with high scores versus people with low scores and established the  benchmarks.  People with higher than average scores had these characteristics:

Years of Web Analytics Experience:  5.4
Interprets reports / suggests actions to be taken: 100% of population
Training / Courses in web analytics:  100% of population
Education post High School: 4.8 Years

People with lower than average scores had these characteristics:

Years of Web Analytics Experience:  2.3
Interprets reports / suggests actions to be taken: 50% of population
Training / Courses in web analytics:  63% of population
Education post High School: 3.6 Years

But inside these averages (segmentation!), it gets much more interesting.  Turns out the less experience you have, the more formal education / training helps you get a higher score.  Education could be college / advanced degrees, vendor training, or classes in web analytics / e-commerce.  Logical, and expected.

Not so intuitive was this on the mix of education and experience: when you have a lot of one and little of the other, you tended to get a lower score.  For example, both Ph.D’s with low years experience and people with 10 years experience but lacking education / training tended to get lower scores.  Likewise, people who indicated they “read blogs and books” as the only source of education did not tend to have high scores unless they had a lot of direct web analytics experience.   So somewhere in the middle there is a “magic mix” of experience and education that results in higher scores.

Interestingly, the single most reliable predictor of a higher score on the test was whether or not in the current job the person regularly suggests actions to be taken based on the analysis.  This data point is more subjective than years of education or experience so we did not include it as a requirement to take the Test, but it’s worth mentioning since it aligns closely with the purpose of the test.

In the end, it’s tough to predict tangible business analysis skills based on just education or experience alone, and this is why the Certification Test should be an important tool for people hiring web analysts.

I’ve heard the Test is Difficult to Pass; can you Explain Why?

In short, because we are a young industry and people tend to have narrow experience relative to the scope of the topic.

You can be an expert in e-mail and Display analytics and still not pass the test because you don’t know enough yet about PPC analysis or Optimizing Web Sites.  You don’t have to be an expert at everything to pass the Test, but you do need to have some knowledge across the entire scope of web analytics to get a high score.  See the Knowledge Required for Certification document for an overview of topics.

That said, I’m sure many of you have been faced before with challenges you did not understand or have any experience with – and then you figured out how to produce insight.  That brainset is precisely what the WAA is testing for.  So if you can take what you know from e-mail analysis and use it to figure out a question about PPC analysis, you could answer the PPC question correctly.  Do that enough times across the different knowledge areas and you could pass the Test, because you essentially demonstrated the ability to think analytically – the objective of the Test.

In opposition to that scenario, blindly following best practices in any knowledge area without recognition of the changes in approach a particular business situation or model might require means you probably will not pass the Test; you will need the capacity to modify your thinking based on the business goals presented.  Example: the correct answer for the publishing model may not be the correct answer for the commerce model.

How Do I Decide if I Should Take the Test?

Honestly, I personally think the Certification has much more value to people who are in the earlier stages of their web analytics  career.   Let’s say you have the same training and read the same books as a lot of other folks.  And you are trying to establish yourself as a person who can create business value but don’t have the resume to back that position up quite yet.  Passing the Certification Test could give you the edge you need to make things happen faster for you.

Conversely, if you have an awesome resume of accomplishments and references for those deeds, then why would you need the additional “proof” the Certification Test provides?  Plus, experienced people often specialize to distinguish themselves from the crowd, and a Test across the universe of Web Analytics would not be particularly relevant.

So I’d expect the majority of people taking the Certification Test to be say 3 – 4 years into their WA careers, or perhaps  earlier if they have been focused on WA and exposed to the right training or experience environments when doing the actual work.

The above is from the perspective of an individual.  However, an agency, consultancy, or service provider might decide having their analysts Certified (including senior people) creates a competitive advantage in their particular space.  Companies looking outside for analytics help may feel more comfortable hiring a resource with WAA Certified talent on staff.

Are there any Questions?

Feel free to ask about anything,  and please see the WAA FAQ for questions on execution details.

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Inside WAA Certification: Any Questions?

You may recall I wrote last year about the explicit or implicit pressure put on Analysts to “torture the data” into analysis with a favorable outcome.  In a piece called Analyze, Not Justify, I described how by my count, about 50% or so of the analysts in a large conference room admitted to receiving this kind of pressure at one time or another.

Since then, I have been on somewhat of a personal mission to try to unearth more about this situation.  And it seems like the problem is getting worse, not better.

I have a theory about why this situation might be worsening.

Companies that were early to adopt web analytics were likely to already have a proper analytical culture.  You can’t put pressure on an analyst to torture data  in a company with this kind of culture – the analyst simply will not sit still for it.  The incident will be reported to senior management, and the source of “pressure” fired.  That’s all there is to it.

However, what we could be seeing now is this: as #measure adoption expands, we find the tools in more companies lacking a proper analytical culture, so the incidents of pressure to torture begin to expand.  And not just pressure to torture, but pressure to conceal, as I heard from several web analysts recently.

One bright young analyst went “beyond the call of duty” on his analytical project.  The analyst gathered relevant data not just from the WA tool, but from Finance, Customer Service – all around the company.  The report painted a detailed picture of cost to acquire customers through various methods and campaigns, and was presented to the head of Marketing – also the analyst’s boss.

The analyst was told under no circumstances was this report to ever be produced again.  Further, the analyst was told to destroy any “evidence” this project / report ever existed.  And finally, the analyst would now be required to send all analysis through the boss first before anybody else sees it.

That’s shameful behavior for an exec.  And apparently, this kind of thing is happening more and more often.  I’ve heard plenty of “if we want your opinion, we’ll ask for it” stories, but this is the first time I’ve heard so many stories about concealing results.

Here’s a scary thought: what if the stories about web analytics not driving business value are primarily concealment stories?   What if the tool / analysts actually did provide value, which was then hidden from Senior Management?

My concern about this issue is wider than screwed up company culture and management.  What I’m more concerned about is screwed up people, analysts who may come to think this kind of behavior is normal and just part of being an analyst.

This matters because as this new generation of analysts moves to other companies and throughout the ecosystem, these pressure to torture situations could become “accepted” and even spread as “part of the game”.

It is never, ever OK to manipulate or hide the results of an analysis.  It’s not part of the job.  The role of an analyst is to analyze, not justify or conceal bad news.

Now, I realize some folks are thinking, “Yea, that’s great Jim, I’ll just get myself fired by being an analytical hero”.

I’m not saying you should respond to data torture pressure by falling on your analytical sword.  What I am saying is you – and management – need to know this kind of pressure from a superior is shameful, not a “normal” part of being an analyst.  And as soon as you can, you should get a job somewhere people respect your professional opinions.  Don’t have to agree; but must respect.

Like the company you work for?  Ask a buddy in Finance if they could use a web analyst.  Pretty sure Finance would be interested in fully-loaded cost to acquire new customers by source!

What really troubles me about this situation is it’s rarely ever talked about, so could be worse than people might think.  At the very least, Senior Management should know about the potential for this to happen and lay down some rules.  Perhaps even seek some cultural guidance on this topic (here’s a start – Fear of Analytics).

So, I want to put this message out there, perhaps create a resource for people who are looking for information on this topic.  It would be great to have examples so managers can understand and be on the lookout for these situations.  Plus, I’m sure there are some terrific stories out there about either giving in to the torture pressure or resisting it!

What about you?  Were you ever pressured to torture the data?  What happened?  Did you comply?  How did things come out?  Tell us with a Comment.  Feel free to post anonymously, leave out company names.

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Tortured Data – and Analysts

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: Thank you for your recent article about Control Groups.  Our organization launched an online distance learning program this past August, and I’ve just completed some student behavior analysis for this past semester.

Using weekly RF-Scores based on Recently and Frequently they’ve logged in to courses within the previous three weeks, I’m able to assess their “Risk Level”– how likely they are to stop using the program.  We had a percentage who discontinued the program, but in retrospect, their login behavior and changes in their login behavior gave strong indication they were having trouble before they completely stopped using it.

A: Fantastic!  I have spoken with numerous online educators about this application of Recency – Frequency modeling, as well online research subscriptions, a similar behavioral model.  All reported great results predicting student / subscriber defection rates.

Q: I’m preparing to propose a program for the upcoming semester where we contact students by email and / or phone when their login behavior gives indication that they’re having trouble.  My hope is that by proactively contacting these students, we can resolve issues or provide assistance before things escalate to the point they defect completely.

A: Absolutely, the yield (% students / revenue retained) on a project like this should be excellent.  Plus, you will end up learning a lot about “why”, which will lead to better executions of the “potential dropout” program the more you test it.

Q: However, in light of your newsletter, I realized that we should probably have a control group with whom we do NOTHING (just as we did this past semester) in order to prove the effectiveness (or not) of the program.

A: Correct.  Otherwise, you won’t be able to make a valid claim to the “saved students”. People can always argue a variety of other factors were in play – seasonality, topic, course sequence, etc.

Q: Since the actual number of students is confidential, can you please tell me what percentage you would use for a control group if we had 400, 800, 1200, 1600, 2000, 3500, or 5000 students?  You mentioned 10% in your newsletter, but the population you were referring to exceeded millions.

A: Well, there are online calculators you can use confidentially, example right here.

If you don’t understand the variables they are asking for, explanations at bottom of page, though this is very simple – what is confidence level and interval plus population size.

Q: Our population is MUCH smaller, and each customer is therefore even more critical.  I don’t want to recommend an unnecessarily large control group that would prevent us from retaining future students when we could see they were having trouble.

I suspect that our defection rates will be lower 2nd semester than 1st since students should be beyond the “learning curve,” so I don’t think we can justly say that the program alone is the reason for lower defection rates if we don’t use a control group.

A: Yes, well, this desire to “get as much test as we can” was the main point discussed in the newsletter.  And that’s the challenge with very small populations – to hit statistical confidence levels at say population = 500, you need over 300 or so in control.

Not so great.

So we go back to the question of company culture and how intuitively confident people will be with the results.  Do they in fact need true statistical significance for a program like this?

There is a way around the significance issue – repetition. The stats part of this is all about the “likelihood you get the same results again” – real important for drug testing, not so much for 500 folks in a marketing program.

The question you need to ask: do you really need “prediction”?  Or does prediction just make the whole test more complex and expensive than it’s worth?  What if you repeated the test a couple of times and got roughly the same results, is that “proof”?

Here is what I might do.  I would ask whoever needs to believe in the results of this test a question like this:

“Let’s say we took a random 20% sample of the students and excluded them from the marketing.  We apply the marketing to the other 80% and their retention rate is 15% higher than the 20% who had no marketing. We do this test 2 more times and the retention rate of students in the test is 13% and 17% higher than the students in the 20% who do not receive the marketing.  Would you at that point believe that without question, the marketing drives at least a 13% improvement in retention among students?”

Do you see where I’m headed with this?  The more times you repeat the test, the more confident you will be in the results – regardless of sample sizes and statistical mumbo jumbo. At some point, the reality of the differences between test and control performance has to be accepted.  It may help to define up front how many repetitions the “boss” needs.

There are two clues to help you evaluate the validity of your results / how many times you need to repeat the test to be “confident”.

One clue is the variability of the results – the more inconsistent the results are, the more likely the data is “noisy” and the more times you need to repeat the test to be confident.

If the spreads between test and control for the first 3 tests are 20%, 5%, and 10%, then you’ll need more repetitions of the test to get a good feeling for the actual impact.  If the results tend to cluster as in the example above (15%, 13%, 17%) then you can be more confident earlier in the test series the actual impact is somewhere around 15%.

The other clue is in the “spread” between test and control.  If the spread is consistently  ”wide”, say +10% (or more), this provides additional confidence a positive impact is being made.  The result over a series of tests may not actually be +10% (confirm by repeating the test), but it’s more likely to be positive.  If you consistently get a spread more like 1% or 2%, it’s more likely the actual result could be zero or negative and you need to keep repeating the test to gain confidence you have a positive result.

In the end, you may not want or be able to repeat the test enough times to know with statistical confidence what the result is.  But if the spread between test and control is wide and consistent, and the cost relative to the benefit is small, then does it really matter if there is statistical confidence?

For example, if you can make the statement you’re confident the program generates at least $10 in profit for each $1 invested, does it really matter if the statistically confident  number is $11 or $12 profit for $1 in cost?  We’re doing Marketing here, not drug testing.  There is an opportunity cost (profit left on the table) to not rolling out a program based on a test with results like this; rather than repeat the test to death just to be more confident I’d roll it out and continue to monitor the results.

One more tip, on this idea of sequencing / semesters / experience with the program.

There is no doubt in my mind that 2nd semester students would have what is called a “survivor bias” and be less likely to drop out; you will get the best performance in a program like this with 1st semester students.  So if at all possible, run the test / control on only 1st semester students , or segment by semester.

But, just because you run it on only 1st semester students does not mean you don’t have an effect in 2nd semester.  Continue to follow test and control into 2nd, 3rd, 4th semesters and you may see the dropout rate of the original 1st semester group continue to widen versus control.

This is not only great for the profitability of the initial 1st semester program but also provides you the baseline you have to beat (control) for those 2nd, 3rd, 4th semesters.  When you decide to see if you can have an additional effect by intervening in those periods, you’ll have 2 groups: those affected by Marketing in the 1st semester, and those new to any Marketing intervention.

My guess: a 1st semester intervention will have tremendous impact, both then and throughout the 4th.  The impact of intervention at each subsequent semester will diminish compared with acting in 1st semester, as will the “tail” value created over the student life, since the number of months left in the student life is shrinking each semester.

Hope that helps!

Jim

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Control Groups in Small Populations