Has to be There

I find it really interesting that whenever there is a discussion of measuring the value of social media, there’s such a bias towards believing there is value in social beyond what can be properly measured.  See the comments following this post by Avinash for a good example.  Speculation is fine, but the confidence being expressed that a new tool or method will uncover a treasure trove of social media value seems un-scientific (as in scientific method) at best.

I don’t doubt there is some value in social media beyond what can be measured, as this has been the case for all marketing since marketing measurement began.  These measurement problems are not new to social either:  Marketing value created is often situational, it depends on the business model and environment.  What works in one situation may not work in another.

For example:

To small local businesses who do no other form of advertising, there is a huge amount of relative value to using social media, versus no advertising at all.  Social advertising is much better than none, and since it’s free, the incremental value created by (properly) using social is huge.  It’s also really easy to measure the impact and true value, since the baseline control is “no advertising”.  Lift, or actual net marketing performance, can be pretty obvious in his case.

On the other hand, many companies are running lots of advertising designed to create awareness, and the incremental value of social as a “media” may be close to zero for these companies, or at least less than the cost to analyze the true value of it.  Possible explanation:  Social events such as “Likes” or comments are simply representations or affirmations of awareness already created by other media, so by themselves, create little value.  In other words, events such as Likes might track the value of other media spending, but may not create much additional marketing value.

Why is this plausible?  It seems unlikely a person would “Like” a brand or product if they have not already experienced it, and are already a fan.  This means in the vast majority of cases, little incremental awareness / acquisition is created.  If this case is not true, if people “Like” a company even though they have no reason to (paid to Like?), then the problems with social marketing analysis go way beyond tools – the concept and data driving the analysis itself is flawed.

But if Like really means Like, the number of Likes or any other similar social events do not have as much to do with awareness as they do with the size of a loyal customer base, and are much more aligned with tracking the success of other awareness / acquisition campaigns.

Looking for Love in All the Wrong Places?

That all said, I believe there is some value being created in the acquisition / awareness area from social.  The problem seems to be this value, when measured, is quite a bit less than everyone expects.  So “the hunt for social value” seems never ending, with speculation and measurements contrived from thin air immensely  popular.  This missing value just has to be there, right?

The core problem is an old one: online value measurement definitions are all over the map, so it’s easy to claim value was created by simply inventing a new way to measure success.  I can’t wait for the day when established test and measurement standards (like using control groups) are adopted in the online space.

Meanwhile, I think it’s quite possible if people keep looking in the awareness / acquisition area, the value of social “beyond what can now be measured”, in many cases, is probably less than the cost of actually measuring it.

Why this approach?  Based on my experience, People are Social, Media are not.  So if you want to derive social value, you use people metrics, not media metrics.

Using this approach, I have unbridled optimism for the value of social.

But I won’t go as far as insisting value is there without measuring it properly first.  Because that’s not how science works.

See ya at eMetrics NYC!

P.S.  There’s lots of real experimental science out there on the effects of social media in the marketing space, have you reviewed it?

You will find this material to be a treasure trove of new ideas and proper methods worth pursuing in the social measurement space, examples here, here, here, here.  Get yourself a subscription to Marketing Science or if you are a WAA member, you can request copies of these fully documented social measurement experiments.

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

“Missing” Social Media Value

Before every eMetrics (I’ll be in San Fran teaching Basecamp, at the Gala, etc.), I try to ask myself, what is the most critical issue facing the web analyst community right now?  Then, at the show, I ask everyone I run into what they think about this issue.

There’s lots of issues to choose from.  Career path I think is a big area of discussion, given the mergers in the space and trend towards outsourcing.  Then there’s the “we don’t get no respect” thing; senior management doesn’t seem to listen / understand / act on the information provided.  And one of my favorites from the past is still out there, data torture - people being pressured to manipulate data to reach a predetermined analytical outcome.

But seems to me, more important at this juncture is trying to resolve why there is so much written about the importance of “the customer” but very little measurement at the customer level.  Think about it.  Customer experience, customer centricity, the entire social thing, it’s all about customers.

But when folks wants to trot out “proof” that this or that approach is the road to the promised land, they analyze impressions, visits, clicks, etc.  Visitor-level stuff.  Does that seem like the correct approach to you?  Seems to me, if you want to provide knowledge about customers, you should measure customers.

One thing we know is customers do express behaviors through a web interface that are not relevant to the future behavior and value of the customer.  One of the earliest and most widely publicized incidents of this was with Amazon gift purchases.  People went on and on about buying a gift from Amazon unrelated to their interests yet having that category Marketed to them relentlessly over time, even though they never purchased from the category again.

This problem was eventually solved by Amazon using Recency, a classic customer behavior metric – only more Recent behavior was used to make suggestions.  Recency is predictive; and lack of behavior is often just as important, if not  more important, than expressed behavior when trying to understand customers.  Unfortunately, most web analysts are trained to look for expressed behavior, not the lack of behavior.

Further, just because an event of some kind happens in the stream of web activity does not mean the event had any affect on the behavior of the customer.  Display impressions, searches, social interactions, all of it – how can you tell whether the event had any effect on the customer at all?  The only way is to measure at the customer level, for example, comparing the behavior of customers who were exposed to the events with customers who were not exposed.  Or, modeling different mixes of events against customer behavior over time, a “marketing mix” model of sorts, to stretch the idea.

Now some people are going to say. “But Jim, we don’t have web tool access to this data!” or “We don’t pass web data to the back end” and all manner of other related excuses, to which I would say,

“Where is your curiosity?”

Clearly, a unified database is best.  But just because your company can’t afford an advanced WA tool doesn’t mean you can’t do this.

I mean seriously, get a dump from the order management system into a spreadsheet.  Run a query against the CRM database.  Look up individual cases in the customer service or lead management systems.  This the way analysts make breakthroughs, how  business cases are built.  If key web data (campaign codes, logins, etc.) doesn’t make it into the back end, why?  If form data crosses over, how hard could it be to send a campaign code, login, or other critical data?  With proof, then pitch the advanced WA tool, or systems, processes, people, whatever you need to make it easier to analyze customer level data.

OK, so let’s hear all the reasons why it’s fine to draw customer-level conclusions using visit-level data, or why you can’t do the above, which I’m sure will include some of the following:

1.  My boss doesn’t care about customer-level data, ignorance is bliss, pseudo-analysis is OK

2.  I’m too busy learning very little about a lot of things instead of going deep on the most important stuff

3.  Shiny objects rule, so see #2 above

4.  I’m a web analyst, back-end data is not my thing

Other reasons?  What do you think?

Do you see the hypocrisy in claiming to understand customer behavior based on visit behavior?

Let’s talk about this at eMetrics San Fran…and Toronto too.

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

All Talk, No #Measure

Just because you pay $5 / CPM for impressions, does not mean they are worth $5 / CPM, does it?  Do people really still have that kind of mentality?  Is the price of the media equivalent to its value?

For example, I’m sure you have heard of multi-million dollar campaigns that generate very little lift in sales.  Happens frequently in fast food, for example.  What is the value of that media?  Is it the millions paid?

What really blows my mind about this approach is it’s so offline, so old school PR. Do the folks who put forth these kinds of models believe nothing has changed in 50 years?  What happened to the whole rap of online being “different”, that you can’t measure it like offline, blah blah.

Except when it’s convenient to do so?

If you want to know the value of a Facebook fan, why not measure the value of a Facebook fan?  Because it’s hard, and would require organizational discipline?  Too bad.   Substituting the kind of models used in the example above for actually measuring the value of a Facebook fan is misleading at the very best.

Make sense?  If you’re with me on this line of thought, let’s not stop here.  We should go ahead and question the value of awareness.

Now comes a better view, but likewise,  just because an event happens does not mean it has value or contributes value.  Looking at the recent post Social media and SEO massively undervalued: study we see a great data collection effort through TagMan but a similar premature jump as above:  that because an event occurs, it somehow must contribute value to the final outcome.  Again, this is a very old-school idea being applied to an environment where there really is no need to guess; set up a test and measure it.

I realize people get excited by the potential of new applications and tools, but have to wonder why folks are so willing to throw logic out the window and “find an answer” even if they have to torture the data to do so.  In many cases the reason is promotional, to sell a product or service, and hopefully this is pretty transparent to the reader.

One of the big problems at the root of all this is the lack of a common value reference point.  In other words, a standard that can be applied to compare the relative value of impressions, events, touches, opens, clicks, and so forth.

This standard exists, it’s called a controlled test.  In academic environments, where all the studies, results, and conclusions are peer-reviewed before they are published, it’s the gold standard for determining “the value of”.  This is a particularly important concept when you are dealing with interactivity; the results of controlled tests can be surprisingly different from common perceptions.

Perhaps it’s time for this community to require (OK, at least ask for?) the same level of transparency.  Count me in.

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

But What is an Impression Worth?

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: First off, I very much appreciate you sharing all this wonderful content on your blog and conferences such as eMetrics.

A: Thanks for that!

Q: My question is a simple one, but I think the answer may be hard: When does a visitor “need” a coupon?  *Need* defined as: visitor would not have placed an order unless presented with the coupon.

A: Hmmm…methinks we’re going to have to define a few concepts and be clear on the goals to make sure we are nailing this down… visitor versus customer, sales versus profit, etc.  In other words, answer is not hard, but could be complex without defining context.

Q: It’s still a mystery to me why so many retailers seem more than willing to hand over all their margins to Groupon or give coupons to basically all visitors.  I am curious whether you would approach this question using  observational data (eg web analytics) or experiments (eg AB testing), or both.

A: Right – is a mystery to me too!

There are certain situations where this approach might be appropriate, but the problem with much web “marketing” (which often is really just advertising without much thought about marketing) is often there is success in a narrow or special situation.  Then the pundits jump on and say “if you’re not doing this you are stupid”, regardless of the business situation and / or without recognizing the special circumstances that are driving success.  This is all the real Marketing stuff people leave out; understanding why it works, under what circumstances, for which segments, involving which products.

When you don’t really understand what is happening and why, no learning takes place.  When no learning takes place, everything is a “new idea” and people are surprised when the outcome is different.

So, for example, if I was launching a brand new service business (restaurant) or a new product that is complex and won’t sell without trial (like yogurt that “naturally regulates your digestive system”), Groupon might be a slam dunk for a product launch.  No surprise here; coupons are often used to drive trial in product launches because the need is to reduce price resistance and drive sampling.

On the other hand, if my product or company is well known and I have tons of loyal customers, Groupon could generally be a financial disaster if you care about profits.  But if all you care about is response to the coupon, it could be a great success!  Because tons of people who would have bought at full price anyway get a huge discount and you get to sell the product below cost.  Awesome!

What do I mean by this, how can you have high response and low or negative profits?

Here is what I have seen over the years: whenever response rates are abnormally high, it means you have a high percentage of responders who would have bought anyway without the coupon.  This is seen over and over in database marketing, online and offline.  From a financial perspective, it means you have probably given up the coupon value with no benefit, a so called “subsidy cost”.

How do you prove this is happening in a promotion?  If you want to really look into and prove these effects, first examine the percentage of response that is from current customers.  If it’s high, that’s the first clue the discount is cannibalistic, not incremental.

If you want to quantify this subsidy cost a bit more is a relatively simple way, take the customer redeemers as a group and look at their average sales for a few months prior to the coupon promotion, during the promotion, and a few months after the promotion.  Often what  you will see is their spending behavior changed very little during the promotion.

For example, let’s say the coupon is 50% off.  The monthly net spending sequence over time might look similar to this:

2 months prior: $100
1 month prior: $100
Promotion month: $50
1 month after: $100
2 months after: $100

This shows the customers redeeming the 50% off coupon did not change their behavior at all; they simply took the discount and bought what they would buy anyway.  Meaning, the coupon cost is a real cost to the bottom line with no offsetting incremental profit.

Bottom line, for every response you lost $50 in sales plus the cost of the campaign, even though you had tons of responses and sales from those responses.  If it was a large campaign, your overall sales for the month net of discounts probably dropped.  Financially, if your cost of goods is 50%, you gave up $25 in profit for every response, minus the per response cost of the campaign.

And, the above behavior is most likely to occur with best, most active customers!  Across all redeemers, you might get $60 or so instead of $50 during the promotion month, but you are still losing money on every redemption – the higher the response, the more money is lost!

This is one big difference between Advertising and Marketing.  Marketing goes beyond Advertising, wants to understand the relationship of specific products to segments of customers, how pricing and modes of distribution affect this relationship, and the profitability of the relationship.

So, with that backdrop, let’s try the question:

Q: When does a visitor “need” a coupon?

A: If I take your question literally, there is a concept in Marketing called coupon proneness, and it’s the classic definition of “needs a coupon”.  Essentially, it means the more coupons you give people the less likely they are to buy without one. If you can imagine what this looks like over time, it’s margin erosion hell.  It’s taking the example above, where no incremental profits were generated, and ensuring it will happen time and time again.

From a Brand perspective, always offering coupons means you are teaching people your prices are too high, or there is a tangible reason your company has to “beg” for sales (implies poor service or quality).  Either way, the outcome is not so good for Brand trust and any evangelism that might result.

The exception to the above is among the “never pay full price” segment, who don’t buy anything without a discount / coupon.  From this segment, you get the benefit (?) of your coupon offers being spread all over the web, attracting many other “never pay full price” customers who generally have negative net values to the company.  Great, huh?

The end result of this pattern is horrible customer loyalty, margin erosion among current customers, and lots of new customers that are 1x buyers.  This means you have to spend *even more* on Advertising to constantly chase new 1x customers, while at the same time your margins in the current customer base are being consistently eroded.

Certainly not an optimized system!

Some people will argue the “extra sales” they get are worth the price of encouraging the above behavior.  But there are not too many businesses that put sales in the bank, what they put in the bank are profits.  So this is very short-term thinking and in fact, you find a lot of businesses that follow this model perform very poorly financially.

So, you ask, when would a visitor “not have placed an order unless presented with the coupon”?  The answer is this: when you have “presented” a coupon before, and the more often you have done this, the less likely they are to buy *until* you present one.

Sure, you could use A/B testing, but it’s not hard to guess what you will find – when you present a coupon, more people buy.  Duh, that’s Advertising, right?  But that’s optimizing for conversion, not for profits, and conversion can’t be deposited in the bank any more than Sales can be.  You have to go further.

For example, if you had the capability to recognize purchase or visit patterns among visitors, you could segment by these behaviors and present coupons on the site only when they were likely to have an incremental rather than cannibalistic outcome.  For example:

1. A new visitor who becomes a repeat visitor X times but does not buy

2. A current customer who has not purchased in over X weeks

and so forth.  You could test for “X” and optimize for highest profitability if you also ran a “null” control group  – where if A = coupon, B = no coupon.  Then look for incremental sales  behavior or “lift” from those offered the coupon versus those not offered a coupon, and run out the profit and loss.

Of course there are other scenarios, mainly current customer comes to your site because  you sent them a coupon, as opposed to presented one on the site.  Not sure if you were including that in your question, but I took your meaning literally.

The scenario with subsidy costs when sending customers a coupon is basically the same as the example above, except you control which customers get what coupon values or if they get a coupon at all.  More info on executing and measuring in that scenario for customers here, and for an example of a company putting this approach into practice, see here.  You can really drive higher profits by doing this correctly.

Jim

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

When Does a Visitor Need a Coupon?

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: I visited your website because I am trying to understand how to develop a customer LifeTime Value model for the company that I work at.  The reason is we are looking at LTV as a way to standardize the ROI measurement of different customer programs.

Not all of these programs are Marketing, some are Service, and some could be considered “Operations”.  But they all touch the customer, so we were thinking changes in customer value might be a common way to measure and compare the success of these programs.

A: Absolutely!  I just answered a question very much like this the other day, it’s great that people are becoming interested in customer value as the cross-enterprise common denominator for understanding success in any customer program!

If I am the CEO, I control dollars I can invest.  How do I decide where budget is best invested if every silo uses different metrics to prove success?  And even worse, different metrics for success within the same silo?

By establishing changes in customer value as the platform for all customer-related programs to be measured against, everyone is on an equal footing and can “fight” fairly for their share of the budget (or testing?) pie.  By using controlled testing, customers can be exposed to different treatments and lift in value can be compared on an apples to apples basis – even if you are comparing the effect of a Marketing Campaign to changes in the Service Center.

But are you sure you want to use LifeTime Value for this application?

Q: From what you stated on your website, I will not be able to develop a LifeTime Value model unless I understand the customer Lifecycle.  The customer lifecycle is something that I could get a good understanding from using doing a RFM analysis.

My question is, once I complete the RFM analysis, what would be my next steps in developing a customer LifeTime Value model?   At this point in time, the hardest thing that I am trying to wrap my head around are the variables to include in the model.  I visited Arthur Middleton Hughes’ website:

http://www.dbmarketing.com

and he suggests the following variables (download spreadsheet, if interested):

http://www.dbmarketing.com/special_ltv.htm

Jim, could I simply use those variables going forward to calculate the LifeTime Value of a customer at my company?  I would appreciate any assistance you may be able to provide to me on this matter.  Thanks.

A: Well, that’s a big tangle of related issues!    Let’s unpack first, then answer the question.  First, the relationships between these ideas:

Lifetime Value versus Lifecycle – LTV is a number, LifeCycle is a trend over time that contains trigger events.  You don’t need the LifeCycle to develop (calculate) LTV, you need the LifeCycle to most efficiently and profitably act on and manage LTV issues.

RFM versus Lifecycle – RFM is a tactical model that is a “snapshot” of customer state at a point in time, the customer’s likelihood to respond.  Frequently used names for these customer states include active, lapsing, lapsed, defected.   Lifecycle is the “movie” one might put together from these snapshots of RFM states; the migration from one customer state to the next are the Lifecycle trigger points.

Now, let’s make sure we understand each one of the ideas:

LifeTime Value

Strictly speaking, LTV is not a very flexible concept and is best used for determining how much you can spend to acquire a customer and still make a profit.  This is the equation that Mr. Hughes has provided, a man by the way that I have a lot of respect for.  His model is quite detailed and useful for the purpose of finding break-even cost to acquire a customer.

To use Arthur’s LTV model, you have to find historical values and plug them in.  You could assume nothing will change and the LTV of certain segments of past customers will be the same; this is great for “benchmarking”, for example.  However, this approach is not measuring LTV, it’s predicting LTV based on historical data.  This is fine, and a valid method for certain types of analysis.

But, the premise of your question is you will be testing, and testing implies something new will occur.  So while you could use LTV to estimate results, you’d have to wait quite a while to prove the results one way or another.  LTV is really “forensic” in this way – you won’t know the final answer until the customers defect.

You could certainly go back 2 – 5 years after the tests, and prove one group had higher LTV than another, but that’s not typically a very useful approach when doing testing.

RFM (Recency, Frequency, Monetary)

RFM is a predictive model that takes a “snapshot” of the customer base and gives you a score for each customer, a prediction of likelihood to respond relative to all customers.

By itself, RFM doesn’t tell you if you are making money or not.  It is used to classify the “state” of customers at a point in time, usually for targeting purposes – are they active, lapsing, lapsed, defected?  In other words, it’s a customer segmentation tool.

For example, RFM could be used to choose your test and control groups for a campaign using Lift measurement – you would want test and control to have the same range and balance of scores.  In fact, one of the tragic campaign measurement mistakes people often make is not taking into account the likelihood to respond when selecting test and control groups, resulting in biased test results.

Customer LifeCycles

One of the great features of RFM is the idea of “ranking” customers relative to each other; this gives allocation of budget and success measurement a standard to follow.  A single  customer can have many different scores over the course of their LifeTime, with the likelihood to respond the score at a specific time.  In fact, if you looked at RFM scores over time for a single customer, you would have a clear understanding of the LifeCycle of a customer – the most powerful segmentation available in terms of message and offer targeting.

The problem with looking at RFM scores over time is complexity; the beauty of individual customer scores at a single point in time becomes unbearable when you are talking 125 different scores on 50,000 customers over 6 months.  That’s the internal or analytical problem.  Externally, this kind of information is extremely gnarly to present and explain to senior managers, it’s presentation hell.

The way I solve this problem is with a tool I call LifeCycle Grids.  The Grids takes the same fundamental drivers used in the RFM model and instead of ranking, uses thresholds or “hurdles” to classify customer states.  This creates a standardized customer LifeCycle “dashboard” so comparisons of customer value between different segments can be made more easily.  It works for both short and long term observations and is easy to represent either numerically or graphically.  And because it uses finite thresholds for activity rather than ranking, the same calculations that create the dashboard can be used to actually drive or trigger actions.

So the dashboard is actually the controller as well.  This is extremely beneficial in terms of linking presentations, plans, and results. People can literally point to a segment on the LifeCycle framework and say, “Let’s deliver message X to each person from segment Y who enters this cell” and see the results right where they pointed when the dashboard is updated.

Once you test some ideas and find out which approach generates incremental profits for a cell in the Grid, you can automate delivery of the program as customers enter that cell of the Grid.  This is the classic “sense & respond” approach to marketing communication – right message, right person, right time.

The LifeCycle Grids are demonstrated in a lot of detail for different applications in the series here, but probably of most interest to you as it relates to customer analysis, see here.

And now, to answer your question:

Which approach above, if any of these, would be best for standardizing measurement of ROI in widely diverse customer programs?

LTV would be appropriate if what you want to know is breakeven cost to acquire.  Since we are talking about customer programs, I doubt that’s what you want to use.  Plus, if you want a hard number rather than a prediction, you could be waiting a long time for the answer.

RFM is a “snapshot” model and so not really suited to long-term studies of customer value.

Customer Lifecycle models are more likely to be involved in the execution of a program, not the success measurement.  LifeCycle tracking could be (and often is) used to predict the financial success of campaigns before they have run their course, but you’re only predicting success, not delivering numbers into an ROI model the CFO would accept as “fact”.

Answer: None of the above.

What you need is an approach designed for the task, which in this case, is:

Lift Measurement or Near-Term Value

Lift is a measure of the performance of a test group of customers compared with a control group of similar customers who are not exposed to the test.  You can read more about control groups here.  In the analysis of value contributed by each group, many of the same values from Arthur’s LTV model are used – product margin, costs of program, fulfillment costs, payment parameters, etc.  However, if you are talking about a program to existing customers, cost to acquire is probably not relevant, though you might use source (campaign) to segment your test approach.

Lift is typically measured at intervals, say every 30 or 60 days, to see how test versus control populations are tracking, and can continue after the test is over to pick up residual value created in the customer.  However, this is not a Lifetime Value measurement, Lift models measure incremental contribution to LTV created by the Marketing, Service, or Operations program execution.

This means if you get lift from program test versus control, when you go back 2 – 5 years later and measure true rather than predicted LTV – after the customer has defected – you should in fact see the LTV in the test group higher than in the control group, barring any radical downstream difference in customer experience between test and control.  In this way, Lift models are actually predictive of changes in LTV.  That’s why the output of Lift models is sometimes referred to as the measurement of “Near-Term Value” and used much more often than the forensic approach of waiting for customers to defect.

Summary

All the above are core concepts in customer value measurement and management.

LTV is a measurement of net financial value contributed by a customer, and Lift measures  are like a “time slice” of the overall LTV curve.

LifeCycles are a management framework for programs designed to affect LTV, and models using Recency, Frequency, and Monetary are used to look at a “time slice” of the LifeCycle.

LTV can generally be increased in two ways: by creating more value during the existing LifeCycle, or by extending the LifeCycle.  Marketing (including Product) is typically used when doing the first, Service and Operations – customer experience and satisfaction – are largely what affects the second.

So it is completely appropriate to establish a unified approach to the measurement of customer programs intended to increase the value of a customer across all these disciplines, in order to ensure the allocation of  scarce resources to highest and best use.

A great question, and for a great cause!

Jim

Update:

Listrak asked me to do a podcast with them on these and related topics, check it out (MP3 link) here, or see list of all their Email Marketing Today podcasts here (I’m on Episode 42).

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

LTV, RFM, LifeCycles – the Framework

So here, I will attempt to answer other kinds of questions I think people might have based on my discussions with WAA members.

Update: The WAA has answered many Certification questions here.

However, I’m going to approach this topic a bit differently than most of the published documentation – from a Product / Marketing perspective, rather than an Educational / WAA POV.  I can do this because (if you don’t know) I have worn all the hats on this project – developer, marketer, WAA project owner – and I think it might be helpful to tell the business story of the WAA Certification, from the bottom up.

And if you have other questions, feel free to leave them in Comments and I will do my best to answer them!

Where did the idea for Certification come from?

The WAA is a member-driven organization; we listen to the membership and try to accomplish what they would like us to accomplish.  We heard from hiring folks and managers that ”web analysts today know a lot of the buzz words and can follow instructions as far as reporting goes, but what we’d be willing to pay a premium for is web analysts who discover things on their own, who add value in areas we don’t already know about”.

So that’s where WAA Certification came from.  It addresses a specific need identified by members, what came to be known internally as the “Book Smart versus Sherlock Holmes” problem.  Sure, you can read a ton of books or blogs and be a  good web analyst by following best practices.  But so can a lot of other people.  What you need to pass the Certification Test is different; you have to be able to turn data into insight and recommend a best action given the scenario presented.

This is Why the WAA’s Educational efforts lack “tool focus”?

Sure.  And of course, the tool vendors already own that focus, and by definition they have the resources to be much better at tool education / certification than the WAA, so why would the WAA want to compete with the tool vendors in the same space?

Better to add value on the business side, where there is demand we can fill and a lack of trusted resources.  And if you think about it, this approach simply expands the overall WA opportunity.  People who want to become experts on the tool side have a path (through the vendors), and people who want to become experts on the analysis / business side also have a path through the WAA.  And if you want to be a Universal Web Analytics Soldier, I guess you could do both!

Does that mean I can pass the Test with No Tool Knowledge?

Not at all.  The threshold we set is you need to be able to communicate effectively with tool experts to pass the test.  That means you will need to know the basics of how the web works, how the tools accomplish their mission, and know what all the web analytics terms mean.  Example: To pass the Test, you don’t need to know how to write a tag, but you do need to know when a  custom tag  is required and how to communicate your need effectively.

So Marketing people can Pass the Certification Test?  eCommerce Managers?  Usability people?  Media Buyers?  Etc.?

Absolutely, if they are good at transforming the data generated by web analytics tools into business insight AND have broad knowledge across the entire scope of web analytics.

Who Created the Certification Test and How?

About 50 WAA members from all over the world volunteered to take on the task.  We created questions, tested them across different audiences, gathered feedback, rewrote the questions based on the feedback, tested the questions again.  You know, the continuous improvement thing?

If you want to participate in the ongoing process of creating the Certification Exam, there is more info here.  Please note you have to be a member of the WAA to be on any WAA Committee.

Where did the Requirements to take the Test Come From?

From the 4 Test the Test sessions we held at various eMetrics events, where we asked people to volunteer to take the Test.  We looked at the backgrounds of  people with high scores versus people with low scores and established the  benchmarks.  People with higher than average scores had these characteristics:

Years of Web Analytics Experience:  5.4
Interprets reports / suggests actions to be taken: 100% of population
Training / Courses in web analytics:  100% of population
Education post High School: 4.8 Years

People with lower than average scores had these characteristics:

Years of Web Analytics Experience:  2.3
Interprets reports / suggests actions to be taken: 50% of population
Training / Courses in web analytics:  63% of population
Education post High School: 3.6 Years

But inside these averages (segmentation!), it gets much more interesting.  Turns out the less experience you have, the more formal education / training helps you get a higher score.  Education could be college / advanced degrees, vendor training, or classes in web analytics / e-commerce.  Logical, and expected.

Not so intuitive was this on the mix of education and experience: when you have a lot of one and little of the other, you tended to get a lower score.  For example, both Ph.D’s with low years experience and people with 10 years experience but lacking education / training tended to get lower scores.  Likewise, people who indicated they “read blogs and books” as the only source of education did not tend to have high scores unless they had a lot of direct web analytics experience.   So somewhere in the middle there is a “magic mix” of experience and education that results in higher scores.

Interestingly, the single most reliable predictor of a higher score on the test was whether or not in the current job the person regularly suggests actions to be taken based on the analysis.  This data point is more subjective than years of education or experience so we did not include it as a requirement to take the Test, but it’s worth mentioning since it aligns closely with the purpose of the test.

In the end, it’s tough to predict tangible business analysis skills based on just education or experience alone, and this is why the Certification Test should be an important tool for people hiring web analysts.

I’ve heard the Test is Difficult to Pass; can you Explain Why?

In short, because we are a young industry and people tend to have narrow experience relative to the scope of the topic.

You can be an expert in e-mail and Display analytics and still not pass the test because you don’t know enough yet about PPC analysis or Optimizing Web Sites.  You don’t have to be an expert at everything to pass the Test, but you do need to have some knowledge across the entire scope of web analytics to get a high score.  See the Knowledge Required for Certification document for an overview of topics.

That said, I’m sure many of you have been faced before with challenges you did not understand or have any experience with – and then you figured out how to produce insight.  That brainset is precisely what the WAA is testing for.  So if you can take what you know from e-mail analysis and use it to figure out a question about PPC analysis, you could answer the PPC question correctly.  Do that enough times across the different knowledge areas and you could pass the Test, because you essentially demonstrated the ability to think analytically – the objective of the Test.

In opposition to that scenario, blindly following best practices in any knowledge area without recognition of the changes in approach a particular business situation or model might require means you probably will not pass the Test; you will need the capacity to modify your thinking based on the business goals presented.  Example: the correct answer for the publishing model may not be the correct answer for the commerce model.

How Do I Decide if I Should Take the Test?

Honestly, I personally think the Certification has much more value to people who are in the earlier stages of their web analytics  career.   Let’s say you have the same training and read the same books as a lot of other folks.  And you are trying to establish yourself as a person who can create business value but don’t have the resume to back that position up quite yet.  Passing the Certification Test could give you the edge you need to make things happen faster for you.

Conversely, if you have an awesome resume of accomplishments and references for those deeds, then why would you need the additional “proof” the Certification Test provides?  Plus, experienced people often specialize to distinguish themselves from the crowd, and a Test across the universe of Web Analytics would not be particularly relevant.

So I’d expect the majority of people taking the Certification Test to be say 3 – 4 years into their WA careers, or perhaps  earlier if they have been focused on WA and exposed to the right training or experience environments when doing the actual work.

The above is from the perspective of an individual.  However, an agency, consultancy, or service provider might decide having their analysts Certified (including senior people) creates a competitive advantage in their particular space.  Companies looking outside for analytics help may feel more comfortable hiring a resource with WAA Certified talent on staff.

Are there any Questions?

Feel free to ask about anything,  and please see the WAA FAQ for questions on execution details.

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Inside WAA Certification: Any Questions?

You may recall I wrote last year about the explicit or implicit pressure put on Analysts to “torture the data” into analysis with a favorable outcome.  In a piece called Analyze, Not Justify, I described how by my count, about 50% or so of the analysts in a large conference room admitted to receiving this kind of pressure at one time or another.

Since then, I have been on somewhat of a personal mission to try to unearth more about this situation.  And it seems like the problem is getting worse, not better.

I have a theory about why this situation might be worsening.

Companies that were early to adopt web analytics were likely to already have a proper analytical culture.  You can’t put pressure on an analyst to torture data  in a company with this kind of culture – the analyst simply will not sit still for it.  The incident will be reported to senior management, and the source of “pressure” fired.  That’s all there is to it.

However, what we could be seeing now is this: as #measure adoption expands, we find the tools in more companies lacking a proper analytical culture, so the incidents of pressure to torture begin to expand.  And not just pressure to torture, but pressure to conceal, as I heard from several web analysts recently.

One bright young analyst went “beyond the call of duty” on his analytical project.  The analyst gathered relevant data not just from the WA tool, but from Finance, Customer Service – all around the company.  The report painted a detailed picture of cost to acquire customers through various methods and campaigns, and was presented to the head of Marketing – also the analyst’s boss.

The analyst was told under no circumstances was this report to ever be produced again.  Further, the analyst was told to destroy any “evidence” this project / report ever existed.  And finally, the analyst would now be required to send all analysis through the boss first before anybody else sees it.

That’s shameful behavior for an exec.  And apparently, this kind of thing is happening more and more often.  I’ve heard plenty of “if we want your opinion, we’ll ask for it” stories, but this is the first time I’ve heard so many stories about concealing results.

Here’s a scary thought: what if the stories about web analytics not driving business value are primarily concealment stories?   What if the tool / analysts actually did provide value, which was then hidden from Senior Management?

My concern about this issue is wider than screwed up company culture and management.  What I’m more concerned about is screwed up people, analysts who may come to think this kind of behavior is normal and just part of being an analyst.

This matters because as this new generation of analysts moves to other companies and throughout the ecosystem, these pressure to torture situations could become “accepted” and even spread as “part of the game”.

It is never, ever OK to manipulate or hide the results of an analysis.  It’s not part of the job.  The role of an analyst is to analyze, not justify or conceal bad news.

Now, I realize some folks are thinking, “Yea, that’s great Jim, I’ll just get myself fired by being an analytical hero”.

I’m not saying you should respond to data torture pressure by falling on your analytical sword.  What I am saying is you – and management – need to know this kind of pressure from a superior is shameful, not a “normal” part of being an analyst.  And as soon as you can, you should get a job somewhere people respect your professional opinions.  Don’t have to agree; but must respect.

Like the company you work for?  Ask a buddy in Finance if they could use a web analyst.  Pretty sure Finance would be interested in fully-loaded cost to acquire new customers by source!

What really troubles me about this situation is it’s rarely ever talked about, so could be worse than people might think.  At the very least, Senior Management should know about the potential for this to happen and lay down some rules.  Perhaps even seek some cultural guidance on this topic (here’s a start – Fear of Analytics).

So, I want to put this message out there, perhaps create a resource for people who are looking for information on this topic.  It would be great to have examples so managers can understand and be on the lookout for these situations.  Plus, I’m sure there are some terrific stories out there about either giving in to the torture pressure or resisting it!

What about you?  Were you ever pressured to torture the data?  What happened?  Did you comply?  How did things come out?  Tell us with a Comment.  Feel free to post anonymously, leave out company names.

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Tortured Data – and Analysts

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: Thank you for your recent article about Control Groups.  Our organization launched an online distance learning program this past August, and I’ve just completed some student behavior analysis for this past semester.

Using weekly RF-Scores based on Recently and Frequently they’ve logged in to courses within the previous three weeks, I’m able to assess their “Risk Level”– how likely they are to stop using the program.  We had a percentage who discontinued the program, but in retrospect, their login behavior and changes in their login behavior gave strong indication they were having trouble before they completely stopped using it.

A: Fantastic!  I have spoken with numerous online educators about this application of Recency – Frequency modeling, as well online research subscriptions, a similar behavioral model.  All reported great results predicting student / subscriber defection rates.

Q: I’m preparing to propose a program for the upcoming semester where we contact students by email and / or phone when their login behavior gives indication that they’re having trouble.  My hope is that by proactively contacting these students, we can resolve issues or provide assistance before things escalate to the point they defect completely.

A: Absolutely, the yield (% students / revenue retained) on a project like this should be excellent.  Plus, you will end up learning a lot about “why”, which will lead to better executions of the “potential dropout” program the more you test it.

Q: However, in light of your newsletter, I realized that we should probably have a control group with whom we do NOTHING (just as we did this past semester) in order to prove the effectiveness (or not) of the program.

A: Correct.  Otherwise, you won’t be able to make a valid claim to the “saved students”. People can always argue a variety of other factors were in play – seasonality, topic, course sequence, etc.

Q: Since the actual number of students is confidential, can you please tell me what percentage you would use for a control group if we had 400, 800, 1200, 1600, 2000, 3500, or 5000 students?  You mentioned 10% in your newsletter, but the population you were referring to exceeded millions.

A: Well, there are online calculators you can use confidentially, example right here.

If you don’t understand the variables they are asking for, explanations at bottom of page, though this is very simple – what is confidence level and interval plus population size.

Q: Our population is MUCH smaller, and each customer is therefore even more critical.  I don’t want to recommend an unnecessarily large control group that would prevent us from retaining future students when we could see they were having trouble.

I suspect that our defection rates will be lower 2nd semester than 1st since students should be beyond the “learning curve,” so I don’t think we can justly say that the program alone is the reason for lower defection rates if we don’t use a control group.

A: Yes, well, this desire to “get as much test as we can” was the main point discussed in the newsletter.  And that’s the challenge with very small populations – to hit statistical confidence levels at say population = 500, you need over 300 or so in control.

Not so great.

So we go back to the question of company culture and how intuitively confident people will be with the results.  Do they in fact need true statistical significance for a program like this?

There is a way around the significance issue – repetition. The stats part of this is all about the “likelihood you get the same results again” – real important for drug testing, not so much for 500 folks in a marketing program.

The question you need to ask: do you really need “prediction”?  Or does prediction just make the whole test more complex and expensive than it’s worth?  What if you repeated the test a couple of times and got roughly the same results, is that “proof”?

Here is what I might do.  I would ask whoever needs to believe in the results of this test a question like this:

“Let’s say we took a random 20% sample of the students and excluded them from the marketing.  We apply the marketing to the other 80% and their retention rate is 15% higher than the 20% who had no marketing. We do this test 2 more times and the retention rate of students in the test is 13% and 17% higher than the students in the 20% who do not receive the marketing.  Would you at that point believe that without question, the marketing drives at least a 13% improvement in retention among students?”

Do you see where I’m headed with this?  The more times you repeat the test, the more confident you will be in the results – regardless of sample sizes and statistical mumbo jumbo. At some point, the reality of the differences between test and control performance has to be accepted.  It may help to define up front how many repetitions the “boss” needs.

There are two clues to help you evaluate the validity of your results / how many times you need to repeat the test to be “confident”.

One clue is the variability of the results – the more inconsistent the results are, the more likely the data is “noisy” and the more times you need to repeat the test to be confident.

If the spreads between test and control for the first 3 tests are 20%, 5%, and 10%, then you’ll need more repetitions of the test to get a good feeling for the actual impact.  If the results tend to cluster as in the example above (15%, 13%, 17%) then you can be more confident earlier in the test series the actual impact is somewhere around 15%.

The other clue is in the “spread” between test and control.  If the spread is consistently  ”wide”, say +10% (or more), this provides additional confidence a positive impact is being made.  The result over a series of tests may not actually be +10% (confirm by repeating the test), but it’s more likely to be positive.  If you consistently get a spread more like 1% or 2%, it’s more likely the actual result could be zero or negative and you need to keep repeating the test to gain confidence you have a positive result.

In the end, you may not want or be able to repeat the test enough times to know with statistical confidence what the result is.  But if the spread between test and control is wide and consistent, and the cost relative to the benefit is small, then does it really matter if there is statistical confidence?

For example, if you can make the statement you’re confident the program generates at least $10 in profit for each $1 invested, does it really matter if the statistically confident  number is $11 or $12 profit for $1 in cost?  We’re doing Marketing here, not drug testing.  There is an opportunity cost (profit left on the table) to not rolling out a program based on a test with results like this; rather than repeat the test to death just to be more confident I’d roll it out and continue to monitor the results.

One more tip, on this idea of sequencing / semesters / experience with the program.

There is no doubt in my mind that 2nd semester students would have what is called a “survivor bias” and be less likely to drop out; you will get the best performance in a program like this with 1st semester students.  So if at all possible, run the test / control on only 1st semester students , or segment by semester.

But, just because you run it on only 1st semester students does not mean you don’t have an effect in 2nd semester.  Continue to follow test and control into 2nd, 3rd, 4th semesters and you may see the dropout rate of the original 1st semester group continue to widen versus control.

This is not only great for the profitability of the initial 1st semester program but also provides you the baseline you have to beat (control) for those 2nd, 3rd, 4th semesters.  When you decide to see if you can have an additional effect by intervening in those periods, you’ll have 2 groups: those affected by Marketing in the 1st semester, and those new to any Marketing intervention.

My guess: a 1st semester intervention will have tremendous impact, both then and throughout the 4th.  The impact of intervention at each subsequent semester will diminish compared with acting in 1st semester, as will the “tail” value created over the student life, since the number of months left in the student life is shrinking each semester.

Hope that helps!

Jim

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Control Groups in Small Populations

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q:  I am hoping you can help answer a question for our team.  By way of introduction, I am the CEO of XXXX.  We are a specialty retailer / restaurant of gourmet pizza, salads and sandwiches.  We would like to know  restaurant industry averages (pizza industry if possible) for customer retention – What percentage of customers that have ordered once from a particular restaurant order from them a second time?  I am hoping with your years of expertise and harnessing data you may be able to assist us with this question.  Look forward to hearing from you.

A:  Unfortunately, in those said years of experience, I have found little hard information on customer retention rates in QSR and restaurants in general (if anyone has data, please leave in Comments).  It’s just the nature of the business that little hard data, if collected, is stored in such a way that one can aggregate at the customer level.  The high percentage of cash transactions doesn’t help matters much; there’s a lot of data missing.

Over the years, sometimes you see data leak out for tests of loyalty programs, and of course clients sometimes have anecdotal or survey data, but this is not much help in getting to a “true” retention rate.  More often than not you discover serious biases in the way the data was collected so at best, you have a biased view of a narrow segment.  Often what you get is a notion of retention among best customers, or customers willing to sign up for a loyalty card, but not all customers.  And the large “middle” group of customers is where all the Marketing leverage is.

What to do about this predicament?  

There are really two issues in your question; the idea of using industry benchmarks when analyzing customer performance, and the measurement of retention in restaurants.

As far as industry benchmarking, two things:

1.  Annual reports for publicly traded eateries may be of help.  Customer loyalty info may be disclosed in these documents or conference calls with Wall Street.  Still, it will probably be of the quality referenced above – narrow in scope or behaviorally biased.

Sometimes you can put snippets of different conversations into an equation that allows you to guess at repeat purchase rate; hospitality analysts often want to understand repeat behavior and do this kind of forecasting.

2.  Ignore the industry benchmarks.  If you have the capability to track repeat rates, simply establish what they are now and use them as internal benchmarks to not fall below or create programs to improve against them.  

Frankly, I tend to discourage using “industry benchmarks” because the kinds of businesses that can really leverage repeat behavior and retention (customer-centric model) are usually *different* from the industry, so using a benchmark (say, from Domino’s) is probably low-balling your potential.  

Not that Domino’s is a “bad” operation, mind you, but they are what they are, they tend to be more on the operational excellence side of the game than customer intimacy (that’s what we called the customer-centric / social approach back in the early 90’s). 

Product leadership, the 3rd value discipline, is pretty much table stakes for anyone in the restaurant biz, and I assume from your business description you just might consider this a primary focus which you then leverage to create power in the intimacy area.  This is essentially the Apple Strategic model.  If the product is not great, the love will not come.

My point is this: without understanding the value discipline and Strategy of a competitor, you can’t know if any benchmark is something you want to compare to, because the business may have a completely different focus than yours.  Worse, using industry averages simply hides any real information you might gain that is actionable for your business.

For example, even though Walmart and Nieman Marcus are in the same business, I don’t think anyone would say they have the same Marketing Strategy or core value proposition.  Walmart is of course the poster child for operational excellence with the end result being value pricing, which flows to the advertising content.  There’s nothing “wrong” with this approach, it simply is what it is, and customer intimacy / relational / social marketing simply doesn’t really fit here.  You certainly can try to be as intimate as possible; but it must be done within the constraints of the model and not reduce operational excellence.  Importantly, this is a “mass” concept, so Push media is the most effective.

Sam’s Club is an example of how one might accomplish this mix.  A “membership” is certainly more customer intimate and allows customized communication, a key component of customer intimate execution.  Again, this flows into the advertising content.  Sam’s gets to leverage the Walmart infra, so they can at the same time maintain a decent level of operational excellence.  Remains to be seen if they could do so without Walmart.

Nieman Marcus on the other hand uses a customer intimate value proposition, and their execution reflects that.  Value pricing is traded off for a high level of customization and personal service, where repeat business is very important since the number of customers this proposition attracts is smaller than the “mass” approach;  you have fewer, but each more valuable, customers.  In this model, mass media is not very effective because the audience is not mass; instead, you rely on the intimacy to Pull customers in, and much more of the Marketing budget is invested not in Advertising, but on in-store (employees, fixtures, locations) and individual communication. 

This relational or customer intimate model is the root of  ”social marketing” and why any attempt to turn online social activity into some kind of mass media advertising opportunity is a complete Paradox.  A step by step example of optimizing the relationship marketing / social model is here: Marketing Bands Series.  To optimize the social model, you divert Marketing budgets away from Mass Advertising and Push into Pull areas like Usability / Store / Interfaces / Packaging, Customer Service, and Customer Retention.

Given the above, would Nieman Marcus ever consider using Walmart’s customer retention rate as a benchmark?  I think not; this approach would make no sense at all.  The mass model can’t leverage customer retention because it’s not intimate; if you can’t act on the metric, why measure it?  This is not to say Walmart “doesn’t care” about repeat business, of course they do.  But they can’t really lever it because it’s more operationally efficient for them to use the mass approach.

That’s a very long explanation for why I dislike using industry benchmarks but many, many people don’t realize how important this idea is; it’s why on a core business model basis some companies will not be able to realize significant benefits from “going “social”.  So on the whole, I would much rather use internal benchmarks that I can improve on that are aligned with the business drivers and are controllable through my own execution.

From looking at your web site, I’d judge you a Nieman as opposed to a Walmart, so customer retention can be a powerful tool for you.  So let’s talk about measuring retention.

“Retention” is a very time-specific concept – over the course of 3 months?  A year?  Five years?  A 20% retention rate over a 5 year period and a 60% retention rate over a 3 month period might both be stunning achievements, if you know what I mean.

So, if you are able to do the analysis, I would pick some marks – 3 month, 6 month, 1 year, etc. – and see what you get for repeat buyer or retention rates.  The slope of that curve will determine where any danger points are that you might take action on.  

For example, if retention falls dramatically from 3 to 6 months, then you know that you should be watching for people who have not transacted in over 3 months, and for  those people you should craft mail / e-mail promotions designed to bring them back.

As often happens with restaurants, there’s probably a good chance that if the person is still living in the area (more on this below), the reason they are not coming back is probably  controllable – they had a bad experience.  A promotion like “We’ve missed you” or “Give us another chance” that is tightly targeted to known defectors will usually pay back quite handsomely in both the short and long term. Defected customers not only visit once on the promo but also (hopefully) have a better experience and re-engage as a repeat visitor.  If your value prop is customer intimate / social, you absolutely must invest in superior customer experience so repeat experiences are rewarding.

If you see some success with this approach, you could then fine tune the analysis to find out if the dropout has a peak in month 3, 4, or 5.  This fine tunes timing of your drop; the closer you can get to the behavior with the message the more effective the campaign will  be.  There is a “peak profitability” timing in one of these months.  

Then the program can be automated, for example: if we don’t see a transaction from this person for 120 days, drop the message.  This way, you end up mailing every month but the audience is completely different and very highly targeted each and every time.  You will find this “right message, to the right person, at the right time” approach is much more profitable than mailing all customers because it directly leverages the customer intimate value prop.

Speaking of mailing all customers, the people who are still active within this 4 month time frame are probably still loyal and you can improve overall margin by not sending these special promotions to those people until they “slip” out of the 4 month window.  There’s no reason to discount to people who are highly likely to purchase anyway.  This is the Pull part of a relationship or social  execution.  What you should be really concerned about are the people who are dis-engaging, where there has been product or service failure.

In fact, in a relational marketing scenario, there is no real need to market to these people at all, you’re basically “preaching to the choir” (example) and doing so is a waste of resources (and often margin).  You will be far better off taking the money you used to spend marketing to the choir and allocating it to in-store, core value proposition ideas.

Many marketing people (especially of the Push variety) find this difficult to understand, but there no more powerful Marketing tool than your value proposition when communicating to the active customer base.  It’s why they are coming back, your Pull is already strong with them.  Why beat them over the head with messages when they are telling you by continued transacting that they like what you are doing?  Wasteful.  (more detailed example)

Finally, in a location-based scenario such as restaurants (and since you are the CEO and not running a single store), you might consider factoring in local uncontrollable churn into any metrics you create as internal benchmarks.  

Households in different areas have different natural churn (move) rates.  Since you have stores in different states, for example, one would expect a lower retention rate from stores that have a higher natural household churn rate.  These stores might be doing very well with controllable churn (product, service) but without the household churn adjustment, they could be unfairly benchmarked “bad”.  HH churn numbers are generally available free from city / state government or the Census.

Hope that helps!

Jim

Note to blog readers: Do you see the parallels above to a lot of what is going on in online publishing / advertising / marketing?  If not, see Jonathan Mendez’s Reaping the Ads You Sow for a more direct analysis of the same concept online.  The strength of the web is in Pull, in converting demand, not Push or creating it.  Use offline for Push; that’s what it’s good at, and synch the two to optimize the entire Marketing ecosystem.

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Relational vs. Transactional

Do you believe human beings share certain fundamental traits that define “being human”?

If so, do you believe that human beings tend to behave in certain ways under certain circumstances?

If so, do you then believe since human behavior has these tendencies, it can often be predicted?

If so, then do you think perhaps the study of Psychology and Sociology might provide you some clues to creating successful businesses, campaigns, products, and services?  While your friends and competitors are all iterating their way into oblivion?

On the web, time and time again, we see the same themes repeating.  Yet with each introduction of a new technology, these themes tend to be treated like a new discovery, even though the theme has been well established in the past.

Norms of Reciprocity is a constant human theme.  You may know the expression of these norms as ”Sharing”.  Web old timers will probably recognize this idea as “Give, then Take” from the I-Sales discussion list as early as 1995.  In various forms, this theme goes back to the beginning of human history, all the way back to the handshake and other greeting gestures.  This same theme is embedded in countless Religions all over the world: “Do onto others as you would wish them do onto you”.  At least a couple centuries old, this idea.

Norms of Reciprocity simply means this: When you do something nice for a human being, help them in some way, this human tends to feel Gratitude towards ”the doer” and tends to do something nice back.  Gratitude drives the desire to Reciprocate, because it’s just what humans do, it’s normal, a “norm”.

Norms of Reciprocity.

The Gratitude cycle doesn’t depend on what the technology is, or if there is any at all.  If anything, technology simply extends the number of humans you can engage in reciprocal behavior with.

I first heard of this theme back in the 1970’s related to the CB radio communities, and it existed before that in ham radio.  Since then, we have been through Compuserve Forums in the 80’s, message boards as early as 1985 with The Well, then e-mail discussion groups, to hybrids like Yahoo Groups, and on into Social Media. 

And in every case, the same rules of successful interaction within these communities always applied, even though the technology was different.  No matter what communications technology the “community” uses, humans find a way to organize it with certain rules. 

And the primary driver of these rules is always Norms of Reciprocity.  Give, then Take.  The rules of successfully participating in any of these communities have not changed at all.

In fact, these reciprocity norms define the meaning of ”community”.  If a “Give, then Take” attitude is not present in a message to the community, then what you have is a message called Advertising.

Advertising has no “Give”, only “Take”.

What does all this have to do with Marketing?

In mass Advertising, it’s extremely difficult to measure the effects of a campaign at the level of Individuals.  You can measure the effects on an Audience as a whole, but not on Individuals.

But when you can measure the impact on Individuals, as you can in many forms of Direct Marketing and on much of online Advertising, now you have the ability to step through a doorway and take advantage of human behavior, including Norms of Reciprocity.

And I think this is where people are getting stuck, including the proponents of everything Social. 

These folks are trying to use Audience measurement models to define the success of (Social) Campaigns targeted to Individuals.   “Social Media” is an oxymoron; it can’t be Social and Media at the same time.

The bottom line is, if you are going to embrace a two-way Social model in Marketing, you must measure the success of this effort differently.  Impressions, reach, size of audience, none of that matters in a model where Relationships - driven by Reciprocity – are the goal.

The above metrics are one-way, broadcast advertising measures.  If “Social” or “Relationships” are to be Marketing models, what’s needed is a way to measure a 2-way exchange, a Relationship.  If it’s the Relationship that’s important, why would you use a “media metric” to measure success?  What you need is a social metric.  A  measure rooted in Psychology, one that addresses Norms of Reciprocity directly.

The question you are trying to answer in Relationship Marketing is not “how many people did I Reach”?  “Influence” or any version of Reach is a crap metric in a Social model; it’s measurement for the sake of measurement.  If it’s Reach you are pegging to, then you’re not Social, you are Media, you are All Take.  There is no Exchange in Reach; Influence is a Social metric Paradox. 

There’s nothing wrong with being Reach-based entity, but just stop calling it Social.  You’re a broadcast tower, a magazine, a newspaper.  Un-Social; Media.  Personally, I don’t think it’s a very good business model, as I said several years ago, unless it goes hyper-vertical to provide context.  That means admitting the “Social as Media” business is much, much smaller than everyone thinks it is.

But let’s say you truly want to be a Social entity or use Social techniques to faciliate Marketing.  Then the real question you need to answer in this Relationship Marketing scenario is: What is the state of my Relationships – Growing, Strong, Weakening, or Failed? 

Why?

Because unless to can define “state”, your Social Marketing efforts are not actionable and you are simply Media.  What you need to know to make Social Marketing work is this: How likely are people to interact with me in the Future?  Because if you know the answer to that question, then you can take the appropriate action against a Growing, Strong, Weakening, or Failed prospect or customer state.

That’s a Relationship.  It’s about the future, not the past.  It’s about Norms of Reciprocity; what I do for or with you today defines what you are likely to do for or with me in the future.  The past is over with; the most important issue is this: where’s the Relationship going?

The question you need to answer in a Social Marketing scenario is not “did they interact with me”, because that’s in the past and there is no Social Power in the past.  The power of Social, the value of ”Give, Then Take”, is in Tomorrow.  Right?  Potential Value.  How much Reciprocity have I earned, what is the Value of this Gratitude in the Future?

The power of Social is not in how many connections you have.

It’s understanding how to make important connections more valuable.

Fortunately, if you use metrics from Psychology rather than Media, the Value and State of your Relationships – Growing, Strong, Weakening, or Failed - are metrics that are not very difficult to measure (example).

Using these Values and understanding Reciprocity, you can then leverage Gratitude and create campaigns that not only Surprise and Delight customers but make a ton of money at the same time.

When you see how well that works, you will want to start segmenting by Relationship State instead of by demographics or other non-Social “Media Metrics” to increase profits by reducing Relationship Friction.

Once you start seeing the cause and effect of true Social or Relationship Marketing, you might even get good enough to see the value of correcting Relationship mistakes before they happen.

Social = Relationship, Relationship = Psychology, not Media.

If you want to do or be Social, then by all means, get on with it already.  There’s already a Model for all this as it applies to Marketing and this model drives profits.  The measurement of success in Social is not unknown and does not require continued mystical thought grazing.  It simply requires you to decide if you are in fact a Social entity and not in reality a Media outlet with fancy new clothes.

If you are starting up a Social entity, the phrase “Norms of Reciprocity” is your gateway to decades of research and testing on humans as Social animals.  This knowledge could save you years of iteration.

If you are already a functioning Social entity, stop gazing into that navel of yours and start publishing quantifiable Metrics from Psychology and Sociology, not Media.  You’ll soon find out whether that Social thing you are doing has Marketing value to anybody or not.

If you are a Marketer trying to leverage the Social in all of us to create and strengthen Relationships, stop looking at Social like Media and demand your vendors do the same.  

Then everybody can skip the million monkey iteration thing. 

Your thoughts on the above?

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Norms of Reciprocity