Category Archives: Marketing Research

Do NPS / CES Feedback Metrics Predict Retention? Depends…

Survey Says?

Several questions came in on the ability of surveys to predict actual behavior, covered in the post Measuring the $$ Value of Customer Experience (see 2. Data with Surveys). My advice is this: if you are interested in taking action on survey results, make sure to survey specific visitors / people with known behavior if possible, then track subjects over time to see if there is a linkage between survey response and actual behavior.  You should do this at least the first time out for any new type of survey you launch.

Why?  Many times, you will find segments don’t behave as they say they will.  In fact, I have seen quite a few cases where people do the opposite of what was implied from the survey.  This happens particularly frequently with best customers – the specific people you most want to please with modifications to product or process.   So this is important stuff.

You’ve Got Data!

Turns out there’s a new academic (meaning no ax to grind) research study out addressing this area, and it’s especially interesting because the topic of study is ability of customer feedback metrics to predict customer retention.  You know, Net Promoter Score, Customer Effort Score and so forth, as well as standard customer satisfaction efforts like top-2-box.

The authors find the ability of any of one of these metrics to predict customer retention varies dramatically by industry.  In other words, you might want to verify the approach / metric you are using by tying survey response to actual retention behavior over time.

Continue reading Do NPS / CES Feedback Metrics Predict Retention? Depends…

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss

Adoption and Abandonment

Out of the Wharton School we have a nice piece of behavioral research on the effect speed of Adoption has on longer-term commitment.  The article, The Long-term Downside of Overnight Success, describes research finding “the adoption velocity has a negative effect on the cumulative number of adopters”. 

This research dovetails nicely with a lot of the topics discussed here on the blog lately, so I thought I’d use it (with a nod to Godin’s post on Strategy vs. Tactics today) to provide some fodder for thought.

First, the importance of Psychology in Marketing.  So many of the “discoveries” arrived at through  brute force testing of Online Advertising are already well known in the greater discipline of Marketing through Psychology.  For more on this read “The Other 3P’s” and if you’d like to do something about lack of knowledge in this area, make sure to read this comment on source books.

Second, this research is a great example of isolating the true drivers of behavior.  The idea of looking at baby names to isolate the real behavior from “technology and other commercial effects” while including “symbolic meaning about identity” results in a broad, Strategic-level answer to the question, not a Tactical one. 

Why is this important?  It means the results can be applied across a host of different Marketing situations, rather than only a specific one. 

Continue reading Adoption and Abandonment

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss

Marketing Science (Journal)

As I said in the Heavy Lifting post, I think the Web Analytics community is becoming increasingly insular and should be paying more attention to what is going on outside the echo chamber in Marketing Measurement.  I also think the next major leaps forward in #wa are likely to come from examining best practices in other areas of Marketing Measurement and figuring out how they apply to the web.

For example, did you even know there is a peer-reviewed journal called Marketing Science, which calls itself “the premier journal focusing on empirical and theoretical quantitative research in marketing”?

Whoa, say what?

This journal is published by the Institute for Operations Research and the Management Sciences, and articles are the work of premiere researchers in visitor and customer behavior from the best known institutions around the world.  In case you didn’t know, “peer-reviewed” means a bunch of these researchers (not including the authors, of course) have to agree that what you say in your article is logical based on the data, and that any testing you carried out adhered to the most stringent protocols – sampling, stats, test construction, all of it.

And, most mind-blowing of all, they show you the actual math right in the article – the data, variables, formulas, graphs – that lead to the conclusions they formulate in the studies.  You know, like this:

Continue reading Marketing Science (Journal)

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss

Online Ads are Navigation

Open your mind for a minute.

What if what the media / agency complex has been telling you all along about online advertising is not really true.  What if Advertising  – from the end user (visitor) perspective – performs a fundamentally different job online than it does offline?  What if the entire game is different than you think it is?  Might that explain why it’s so difficult to get any agreement on the value of online advertising?

Please bear with me; see if this makes any sense to you.

Offline, it’s important that you remember an ad.  That’s because you are rarely in a position to take advantage of or act on the ad when you are exposed to it – unless you are sitting in front of a computer.  Awareness, Recall, all those nice measurements the offliners do are important for offline Advertising, because the job of offline Advertising is get you to remember it so you can Act on the Advertising when you are in a position to do so.

Online, you can immediately investigate the products or services advertised, get 3rd party opinions, and so forth.  You can convert Awareness to Intent and Desire in a matter of moments, if not take Action as well –  if you are interested in what is being Advertised.

The fundamental answer to every question you have about online advertising might be really simple, if you think this way:

Online Ads are Navigation

They are not Advertising, in the traditional sense of offline Advertising.

Content sources serve the role of traditional Advertising online.

Not the ad itself.

Online, the Web Site is the Ad.

Continue reading Online Ads are Navigation

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss

Offline Path Analysis

It’s always a treat to work with bright, committed people and I’m happy to say this was the case with the folks at the Oriental Institute.  These higher ed environments can be exceedingly complex from a Marketing perspective, and the OI is way up there on the complexity scale.  So much to do, so few resources to do it with.

That said, we came up with a crackerjack plan that should significantly boost paid Membership at the OI without additional time or money resources.  How?  Path Analysis.

Personally, I have never understood why many web analytics folks don’t care for Path Analysis; I can only surmise these folks are simply not doing it correctly.  For one thing, Paths don’t make any sense without the context of a behavioral segmentation – entry page, campaign, etc.  Just like any other web data, Path is useless without segmentation.  Or perhaps these folks don’t know how to interpret the data they see because they can’t survey a Path for the answers. 

Continue reading Offline Path Analysis

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss

Online Stat of the Year?

Over on the Rimm-Kaufman Group blog was a report on what Forrester’s Carrie Johnson had to say at Shop.org’s Marketing Workshop.  There are quite a few interesting tidbits, but here’s the pair that blew me away:

Correlation between Google Gross US Revenues to US E-Commerce Growth: .96.

Correlation with Yahoo Display Ad Sales and US E-Commerce Growth: -.04

Now, I understand that Correlation does not imply Causation but at some level when you get directional spreads like this you have to sit up and take notice.

One explanation is this:  e-Commerce sites do not buy any Display to speak of, but we know that’s not true – don’t we

Other questions:

1.  Another conclusion would be Yahoo matters very little to e-commerce activity.  Sure, less than Google, but to this degree?  If in fact Display enhances Search performance, you would think Yahoo would have more of an effect.  Perhaps folks see Display on Yahoo and then Search on Google?   Wouldn’t that be a trip…

That scenario would really provide a whole new twist on the measurement of view-throughs.

2.  Google gross rev’s include AdSense, of course.  So we’re not really comparing PPC to Display here, though one could argue AdSense is more targeted than Display.  So what we are discussing here is the relevance of ads, not PPC versus Display.

3.  Does Yahoo Display include Travel ads triggered by selection of Location?  Auto ads triggered by selection of Model?  Etc.  Etc.  You could argue those ads are really “Search” if you look at it from a behavioral (customer) perspective.

Sure would like to find the source on this, and see what we are actually talking about here.

Other questions you would ask / data you need to make a judgment on this?  How about wild speculations on what this data means, if anything?

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss

Interview-Podcast w/ Jim Novo

Friend and fellow blogger Alan Rimm-Kaufman spent some of his valuable time asking my opinion on various online marketing issues in a far-ranging interview and podcast.

We met in person for the first time doing a presentation together at the DMA show in Chicago this fall, and because he used to work at Crutchfield – a truly customer-driven remote retailer – we share some experiences and beliefs.

For those of you who might be wondering where a lot of the Marketing Productivity ideas I post here come from, this interview-podcast is probably a pretty good backgrounder.  We talk about a lot of stuff, including:

Monetizing customer experience

Importance of Control Groups / Source Attribution

Multichannel Marketing Strategy

LifeCycle Contact Strategy versus Calendar-based

Retail Business Models / Lab Store

Search box or not? / Serendipity

How to tell if online customers are really engaged – without web analytics

Here’s another link to the Interview-Podcast.  Enjoy! 

That was lots of fun, thanks Allen!

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss

Al Gore & Warren Buffet: Marketing Gurus

Um, following up on the post Research for Press Release, we have this gem from eMarketer and Anderson Analytics, who apparently did not even read the results of the survey they conducted.

Some highlights from this group of “Senior Marketers”:

Most important thing they are concentrating on:

Mastering the Basics

Seems unusual for Senior Marketers, to me.  

My guess: the members of MENG are not Senior Marketers, and should not be referred to as such.  Of course, nobody would pay attention to a press release about a survey on a “bunch of pukes”; this is the Source of Sample problem.

Asking “which demographic segment is most important to target” generically without supplying the product to be marketed is a ridiculous concept.  “Senior Marketers” probably wouldn’t even answer this question.

And the biggest gut-splitter: the list of “Most Important Marketing Gurus” includes Al Gore & Warren Buffet.  Now, these are both smart gents in their own ways but I’m not aware of their status as Marketing Gurus.

Of course, an alternative reality is possible: the members of MENG are Senior Marketers.  If that’s the case, I simply don’t know what to say, other than Marketing has probably already Deconstucted.  Or Imploded.  Or something worse.

You can learn a lot more from this really useful Research for Press Release (RFPR?) piece here.

Comments?

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss

More Tips on Evaluating Research

To continue with this previous post…other things to look for when evaluating research:

Discontinuous Sample – I don’t know if there is a scientific word for this (experts, go ahead and comment if so), but what I am referring to here is the idea of setting out the parameters of a sample and then sneaking in a subset of the sample where the original parameters are no longer true.  This is extremely popular in press about research.

Example:  A statement is made at the beginning of the press release regarding the population surveyed.  Then, without blinking an eye, they start to talk about the participants, leaving you to believe the composition of participants reflects the original population.  In most cases, this is nuts, especially when you are talking about sending an e-mail to 8000 customers and 100 answer the survey. 

Sometimes it works the other way, they will slip in something like, “50% of the participants said the main focus of their business was an e-commerce site”, which does not in any way imply that 50% of the population (4000 of 8000) are in the e-commerce business.  Similarly, if you knew what percent of the 8000 were in the e-commerce business, then you could get some feeling for whether the participant group of 100 was biased towards e-commerce or not.

Especially in press releases, watch out for these closely-worded and often intentional slights of hand describing the actual segments of participants.  They are often written using language that can be defended as a “misunderstanding” and often you can find the true composition of participants in the source documentation to prove your point. 

The response to your digging and questioning of the company putting out the research will likely be something like, “the press misunderstood the study”, but at least you will know what the real definitions of the segments are.

Get the Questions – if a piece of research really seems to be important to your company and you are considering purchasing it, make sure the full report contains all the research questions

I can’t tell you how many times I have matched up the survey data with the sequencing and language of the questions and found bias built right into the survey.  Creating (and administering, for that matter) survey questions and sequencing them is a scientific endeavor all by itself.  There are known pitfalls and ways to do it correctly, and people who do research for a living understand all of this.  It’s very easy to get this part of the exercise wrong and it can fundamentally affect the survey results.

So, in summary, go ahead and “do research” by e-mailing customers or popping up questionnaires, or read about research in the press, but realize there is a whole lot more going on in statistically significant, actionable research than meets the eye, and most of the stuff you read in the press in nothing more than a Focus Group.

Not that there is anything inherently wrong with a Focus Group, as long as you realize that is what you have.

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss

Research for Press Release

I think one of the reasons “research” has become so lax in design and execution is this idea of doing research to drive a press release and news coverage.  Reliable, actionable research is expensive, and if all you really want to do is gin out a bunch of press, why be scientific about it?  Why pay for rigor?  After all, your company is not going to use the research to take action, it’s research for press release.

So here’s a few less scientific but more specific ideas to keep in mind when looking at a press release / news story about the latest “research”, ranked in order of saving your time.  In other words, if you run into a problem with the research at a certain level, don’t bother to look down to the next level – you’re done with your assessment.

Press about Research is Not Research – it’s really a mistake to make any kind of important decision on research without seeing the original source documentation.  For lots of reasons, the press accounts of research output can be selectively blind to the facts of the study. 

If there is no way to access the source research document, I would simply ignore the press account of the research.  Trust me, if the subject / company really had the goods on the topic, they would make the research document available – why wouldn’t they?  Then if / when you get to the research source document, run the numbers a bit for your self to see if they square with the press reports.  If not, you still may learn something – just not what the press report on the research was telling you!

Source of Sample – make sure you understand where the sample came from, and assess the reliability of that source.  Avoid trusting any source where survey participants are “paid to play”.  This PTP “research” is often called a Focus Group and though you can learn something in terms of language and feelings and so forth from a Focus Group, I would never make a strategic decision based on a non-scientific exercise like a Focus Group. 

Go ahead and howl about this last statement Marketers,  I’m not going to argue the fine points of it here, but those wish to post on this topic either way, go ahead.  Please be Less Scientific or More Specific than usual, depending on whether you are a Scientist or a Marketer. 

For a very topical and probably to some folks quite important example of this “source” problem, see Poor Study Results Drive Ad Research Foundation Initiative.  If you want a focus group, do a focus group.  But don’t refer to it as “research” in a scientific way.

Size of Sample – there certainly is a lot of discussion about sample sizes and statistical significance and so forth in web analytics now that those folks have started to enter the more advanced worlds of test design.  Does it surprise you the same holds true for research?  Shouldn’t, it’s just math (I can feel the stat folks shudder.  Take it easy, relax).

Without going all math on this, let’s say someone does a survey of their customers.  The survey was “e-mailed to 8,000 customers” and they get 100 responses to the survey.   I don’t need to calculate anything to understand the sample is probably not representative of the whole, especially given the methodology of “e-mailed our customers”.  Not that a sample of 100 on 8000 is bad, but the way it was sourced is questionable.

What you want to see is something more like “we took a random sample of our customers and 100 interviews were conducted”.  It’s the math thing again.  Responders, by definition, are a biased sample, probably more of a focus group.  This statement is not always true, but is true often enough that you want to verify the responders are representative.  Again, check the research documentation.

OK Jim, so how can political surveys be accurate when they only use 300 or so folks to represent millions of households?  The answer is simple.  They don’t email a bunch of customers or pop-up surveys on a web site.  They design and execute their research according to established scientific principles.  Stated another way, they know exactly and specifically who they are talking to.  That’s because they want the research to be precise and predictive.

How do you know when a survey has been designed and executed properly?  Typically, a confidence interval is stated, as in “results have margin of error +- 5%”.  This generally means you can trust the design and execution of the survey because you can’t get this information without a truly scientific design (Note to self, watch for “fake confidence level info” to be included with future “research for press release” reporting).

More rules for interpreting research

Share:  twittergoogle_plusredditlinkedintumblrmail


Follow:  twitterlinkedinrss