Inside WAA Certification: Any Questions?

The WAA has published a lot of info about the new WAA Certification Exam; you might want to first read the FAQ and take a look at the application information and Exam Handbook for the organizational details, and you can see sample questions from the Test at the bottom of the page here.  But something I can just about guarantee about the Certification – no matter how much info the WAA publishes about it, many people will still have questions!

So here, I will attempt to answer other kinds of questions I think people might have based on my discussions with WAA members.

Update: The WAA has answered many Certification questions here.

However, I’m going to approach this topic a bit differently than most of the published documentation – from a Product / Marketing perspective, rather than an Educational / WAA POV.  I can do this because (if you don’t know) I have worn all the hats on this project – developer, marketer, WAA project owner – and I think it might be helpful to tell the business story of the WAA Certification, from the bottom up.

And if you have other questions, feel free to leave them in Comments and I will do my best to answer them!

Continue reading Inside WAA Certification: Any Questions?

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss

Tortured Data – and Analysts

Fear and Loathing in WA

You may recall I wrote last year about the explicit or implicit pressure put on Analysts to “torture the data” into analysis with a favorable outcome.  In a piece called Analyze, Not Justify, I described how by my count, about 50% or so of the analysts in a large conference room admitted to receiving this kind of pressure at one time or another.

Since then, I have been on somewhat of a personal mission to try to unearth more about this situation.  And it seems like the problem is getting worse, not better.

I have a theory about why this situation might be worsening.

Companies that were early to adopt web analytics were likely to already have a proper analytical culture.  You can’t put pressure on an analyst to torture data  in a company with this kind of culture – the analyst simply will not sit still for it.  The incident will be reported to senior management, and the source of “pressure” fired.  That’s all there is to it.

However, what we could be seeing now is this: as #measure adoption expands, we find the tools in more companies lacking a proper analytical culture, so the incidents of pressure to torture begin to expand.  And not just pressure to torture, but pressure to conceal, as I heard from several web analysts recently.

Continue reading Tortured Data – and Analysts

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss

Control Groups in Small Populations

The following is from the January 2010 Drilling Down Newsletter.  Got a question about Customer Measurement, Management, Valuation, Retention, Loyalty, Defection?  Just ask your question.  Also, feel free to leave a comment and I’ll reply.

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: Thank you for your recent article about Control Groups.  Our organization launched an online distance learning program this past August, and I’ve just completed some student behavior analysis for this past semester.

Using weekly RF-Scores based on Recently and Frequently they’ve logged in to courses within the previous three weeks, I’m able to assess their “Risk Level”– how likely they are to stop using the program.  We had a percentage who discontinued the program, but in retrospect, their login behavior and changes in their login behavior gave strong indication they were having trouble before they completely stopped using it.

A: Fantastic!  I have spoken with numerous online educators about this application of Recency – Frequency modeling, as well online research subscriptions, a similar behavioral model.  All reported great results predicting student / subscriber defection rates.

Q: I’m preparing to propose a program for the upcoming semester where we contact students by email and / or phone when their login behavior gives indication that they’re having trouble.  My hope is that by proactively contacting these students, we can resolve issues or provide assistance before things escalate to the point they defect completely.

A: Absolutely, the yield (% students / revenue retained) on a project like this should be excellent.  Plus, you will end up learning a lot about “why”, which will lead to better executions of the “potential dropout” program the more you test it.

Continue reading Control Groups in Small Populations

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss

Acting on Buyer Engagement

Over the years I’ve argued that there is a single, easy to track metric for buyer engagement – Recency.  Though you can develop really complex models for purchase likelihood, just knowing “weeks since last purchase” gets you a long way to understanding how to optimize Marketing and Service programs for profit.

Which brings me to the latest Marketing Science article I have reviewed for the Web Analytics Association, Dynamic Customer Management and the Value of One-to-One Marketing, where the researchers find “customized promotions yield large increases in revenue and profits relative to uniform promotion policies”.  And what variable is most effective when customizing promotions?

The researchers took 56 weeks of purchase behavior from an online store, and used the first 50 weeks to construct a predictive model of purchase behavior.   Inputs to the model included Price, presence of Banner Ads, 3 types of promotions, order sizes, number of orders, merchandise category, demographics, and weeks since last purchase (Recency).

The last 6 weeks of data were used to test the predictive power of the model, and the answer to which variable is most predictive of purchase is displayed in the chart below, click to enlarge:

Weeks since last purchase dominated the predictive power of the model, controlling not only the Natural purchase rate (labeled Baseline in chart above, people who received no promotions) but the response to all three different types of promotion.

Continue reading Acting on Buyer Engagement

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss

Choosing the Size of Control Groups

The following is from the December 2009 Drilling Down Newsletter.  Got a question about Customer Measurement, Management, Valuation, Retention, Loyalty, Defection?  Just ask your question.  Also, feel free to leave a comment and I’ll reply.

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

 Q:  I am a big fan of your web site and read your Drilling Down book. Great work!

A:  Thanks for the kind words!

Q:  I was wondering if you could help me picking the right control group size for a project of ours?  The population is 25 million telco customers that for which we want to do a long term impact analysis (month by month) in regards to revenue increase versus control group.  The marketing initiatives are mix of retention, lifecycle and tactical/seasonal activities.  We want to measure revenue increase through any of the marketing activities compared to control group.

A:   Great project, this is the kind of idea that can really improve margins if you can find out which specific tactics drop the most profit to the bottom line.

Q:   I have searched the web for some help and found calculators that say: On 25 million and smallest expected uplift of 0.1% and highest likely rate of > 5% the calculator gives 250k (1%).  Is that sufficient to calculate the net impact on the remaining base?  Would be very grateful if you could give me your thoughts.

A:  Well, it could be and might not be…

Continue reading Choosing the Size of Control Groups

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss

Customer Value in the Freemium Model

The following is from the November 2009 Drilling Down Newsletter.  Got a question about Customer Measurement, Management, Valuation, Retention, Loyalty, Defection?  Just ask your question.  Also, feel free to leave a comment and I’ll reply.

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: You kindly clarified a few issues when I was reading Drilling Down earlier this year – so I hope you don’t mind the direct email.

A: Yes, I remember!

I am working for www.XYZ.com, a social networking / virtual world site based abroad but visitors are 85% US.

Our growth up to now has been mainly viral and in the summer we hit 1.2M UVs operating on the Freemium model with only 5% of our registered users converting to paying customers and a significant portion of our revenue coming from ads.  On average our customers are active on the site for something like 4 months making their first purchase around day 28. 

But to take us to the next stage we are embarking on some marketing for the first time using AdWords and various revenue share campaigns, and of course to do this sensibly we need to arrive at a reasonable estimate of LTV.

A: Makes sense!

Q: To calculate an adjusted LTV I removed all customers with a lifetime of less than 4 months but this gives a low estimate as this calculation ignores the bumper summer months and the extra paid for features put in place earlier this year.  Calculating LTV using ARPU and monthly churn (not sure how to calculate this in our environment) gives another different estimate.  Is there any help or advice you could perhaps give us?  If not in the US then perhaps you could recommend somebody abroad – can’t find anything in the literature relevant for start-up like us.

A:  It sounds to me like you’re trying to make this too complicated, at least for the place you are at this time.  Monthly churn and the “28 day” threshold are nice to know on a tactical level, but LTV is more of a Strategic idea that does not necessarily benefit from analysis at that level.  And you may not really want LTV, but a derivative that might be more helpful.

Continue reading Customer Value in the Freemium Model

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss

“X Month” Value

The basic concept of LifeTime Value (LTV) was ably outlined by Seth Godin in a great post here.  If you know the average net value of a customer is $2500 over their “Life”, why would you not spend  $50 (or $200, really) to acquire each one?  As long as you stuck to the model, your company would be insanely profitable over time.

Their are 2 primary challenges to implementing this idea.

1.  “Over time” is a concept many management folks have a hard time embracing; what matters are the profits this year, or this quarter, or this month.  Unless the whole company embraces an “over time” measurement approach it is difficult for Marketers and Analysts to drive towards programs and practices supporting the LTV outcome.

2.  The $2500 is an average figure.  Most customers are worth less; 10% or 20% are worth much more.

Most people I talk to embrace the general idea of LTV models intuitively.  It’s really a cash flow concept, isn’t it?

So Financial people get it right away, and if Marketers could align with it, there would be no conflicts and the Marketing budget becomes virtually unlimited.

In fact, many folks in the PPC world follow just this model – they have unlimited budget as long as each conversion costs no more than “X”.  Because the company knows if it spends no more than X on a conversion, it always makes money.   Marketers and Analysts involved with these “Cost < X” PPC programs love them, because Management loves them. 

And Management loves them, why?  Because the CFO loves these programs  Why?  Because they are based on Cash Flow analysis, which CFO’s understand very, very well.

So then, what will it take to get more acquisition budgets like these Cost < X  PPC programs?  We have to address the two challenges above:

Continue reading “X Month” Value

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss

Member Retention in Professional Orgs

The following is from the October 2009 Drilling Down Newsletter.  Got a question about Customer Measurement, Management, Valuation, Retention, Loyalty, Defection?  Just ask your question.  Also, feel free to leave a comment and I’ll reply.

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: I have recently purchased your book Drilling Down and going through the many interesting concepts.

A: Thanks for that!

Q:  I work for a membership Organization and we would like to conduct some analysis into who we may lose and approach them even before their membership lapses.  But the only problem here is that we carry data only on the purchases made (though many of our members do not purchase our products and stay a member) and web site visits.

A:  Are you *sure* that’s all the data you collect?  I once worked with a professional membership org that thought they only had one data source, but turns out they had 8 – from 8 different areas of the org – that nobody really knew about.

Q:  How do I know if a particular member is going to resign and lapse soon with this limited amount of behavioral data.  Recently it’s been a concern that we are losing members who have been with us for more than 10 years and who are in their mid career profession (aged between 30 to 45) and indicated no specific reason for resignation. 

This has been going on for the last few months and now we would like to strategically target these customers and approach them even before they react negative.  What concepts could help me to do this? Your guidance would be much appreciated.

A:  OK, my answer will be in two sections: if you (hopefully) find you have more data than you think, and if you really don’t have any other data to fall back on.

Continue reading Member Retention in Professional Orgs

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss

Relational vs. Transactional

The following is from the September 2009 Drilling Down Newsletter (original title:  Customer Retention for Restaurants).  Got a question about Customer Measurement, Management, Valuation, Retention, Loyalty, Defection?  Just ask your question.  Also, feel free to leave a comment.

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q:  I am hoping you can help answer a question for our team.  By way of introduction, I am the CEO of XXXX.  We are a specialty retailer / restaurant of gourmet pizza, salads and sandwiches.  We would like to know  restaurant industry averages (pizza industry if possible) for customer retention – What percentage of customers that have ordered once from a particular restaurant order from them a second time?  I am hoping with your years of expertise and harnessing data you may be able to assist us with this question.  Look forward to hearing from you.

A:  Unfortunately, in those said years of experience, I have found little hard information on customer retention rates in QSR and restaurants in general (if anyone has data, please leave in Comments).  It’s just the nature of the business that little hard data, if collected, is stored in such a way that one can aggregate at the customer level.  The high percentage of cash transactions doesn’t help matters much; there’s a lot of data missing.

Over the years, sometimes you see data leak out for tests of loyalty programs, and of course clients sometimes have anecdotal or survey data, but this is not much help in getting to a “true” retention rate.  More often than not you discover serious biases in the way the data was collected so at best, you have a biased view of a narrow segment.  Often what you get is a notion of retention among best customers, or customers willing to sign up for a loyalty card, but not all customers.  And the large “middle” group of customers is where all the Marketing leverage is.

What to do about this predicament?  

There are really two issues in your question; the idea of using industry benchmarks when analyzing customer performance, and the measurement of retention in restaurants.

Continue reading Relational vs. Transactional

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss

Awareness versus Persuasion

In the early days of Home Shopping Network (live TV, not online), we were doing some ethnographic research and started to find “physical clusters” of customers – neighbors or people who worked together.  For example, one of these groups was nurses at hospitals,  especially nurses  who worked the night shift.

We looked for the most active member of the cluster (our “thought leader”) and asked them if they would help us with a “member get a member” program.  Would they be willing to distribute discount coupons to their friends, especially ones who were not already customers?  Time after time, the answer was:

“Honey, all my friends are already customers of yours”.

We launched the program anyway, because it was a pet project from upstairs  – I was a junior marketer at that point so I couldn’t kill it ;)  The program never, ever worked, no matter how hard we tried.  It generated very few new customers while giving lots of discounts to people who were already active buyers.  Basically,  the cost of those discounts overwhelmed the value of the new customers generated.

Apparently a similar thing happens online with Social marketing.

As part of a WAA program that reviews academic research for WAA members, I was able to take a look at a paper titled:  Firm-Created Word-of-Mouth Communication: Evidence from a Field Test by David Godes and Dina Mayzlin.

Continue reading Awareness versus Persuasion

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss