Category Archives: Analytical Culture

More Tips on Evaluating Research

Analytical Culture, Analytics Education, Marketing Research

To continue with this previous post…other things to look for when evaluating research:

Discontinuous Sample – I don’t know if there is a scientific word for this (experts, go ahead and comment if so), but what I am referring to here is the idea of setting out the parameters of a sample and then sneaking in a subset of the sample where the original parameters are no longer true.  This is extremely popular in press about research.

Example:  A statement is made at the beginning of the press release regarding the population surveyed.  Then, without blinking an eye, they start to talk about the participants, leaving you to believe the composition of participants reflects the original population.  In most cases, this is nuts, especially when you are talking about sending an e-mail to 8000 customers and 100 answer the survey. 

Sometimes it works the other way, they will slip in something like, “50% of the participants said the main focus of their business was an e-commerce site”, which does not in any way imply that 50% of the population (4000 of 8000) are in the e-commerce business.  Similarly, if you knew what percent of the 8000 were in the e-commerce business, then you could get some feeling for whether the participant group of 100 was biased towards e-commerce or not.

Especially in press releases, watch out for these closely-worded and often intentional slights of hand describing the actual segments of participants.  They are often written using language that can be defended as a “misunderstanding” and often you can find the true composition of participants in the source documentation to prove your point. 

The response to your digging and questioning of the company putting out the research will likely be something like, “the press misunderstood the study”, but at least you will know what the real definitions of the segments are.

Get the Questions – if a piece of research really seems to be important to your company and you are considering purchasing it, make sure the full report contains all the research questions. 

I can’t tell you how many times I have matched up the survey data with the sequencing and language of the questions and found bias built right into the survey.  Creating (and administering, for that matter) survey questions and sequencing them is a scientific endeavor all by itself.  There are known pitfalls and ways to do it correctly, and people who do research for a living understand all of this.  It’s very easy to get this part of the exercise wrong and it can fundamentally affect the survey results.

So, in summary, go ahead and “do research” by e-mailing customers or popping up questionnaires, or read about research in the press, but realize there is a whole lot more going on in statistically significant, actionable research than meets the eye, and most of the stuff you read in the press in nothing more than a Focus Group.

Not that there is anything inherently wrong with a Focus Group, as long as you realize that is what you have.

Research for Press Release

Analytical Culture, Analytics Education, Marketing Research

I think one of the reasons “research” has become so lax in design and execution is this idea of doing research to drive a press release and news coverage. Reliable, actionable research is expensive, and if all you really want to do is gin out a bunch of press, why be scientific about it?  Why pay for rigor?  After all, your company is not going to use the research to take action, it’s research for press release.

So here’s a few less scientific but more specific ideas to keep in mind when looking at a press release / news story about the latest “research”, ranked in order of saving your time.  In other words, if you run into a problem with the research at a certain level, don’t bother to look down to the next level – you’re done with your assessment.

Press about Research is Not Research – it’s really a mistake to make any kind of important decision on research without seeing the original source documentation.  For lots of reasons, the press accounts of research output can be selectively blind to the facts of the study. 

If there is no way to access the source research document, I would simply ignore the press account of the research.  Trust me, if the subject / company really had the goods on the topic, they would make the research document available – why wouldn’t they?  Then if / when you get to the research source document, run the numbers a bit for your self to see if they square with the press reports.  If not, you still may learn something – just not what the press report on the research was telling you!

Source of Sample – make sure you understand where the sample came from, and assess the reliability of that source.  Avoid trusting any source where survey participants are “paid to play”.  This PTP “research” is often called a Focus Group and though you can learn something in terms of language and feelings and so forth from a Focus Group, I would never make a strategic decision based on a non-scientific exercise like a Focus Group. 

Go ahead and howl about this last statement Marketers,  I’m not going to argue the fine points of it here, but those wish to post on this topic either way, go ahead.  Please be Less Scientific or More Specific than usual, depending on whether you are a Scientist or a Marketer. 

For a very topical and probably to some folks quite important example of this “source” problem, see Poor Study Results Drive Ad Research Foundation Initiative.  If you want a focus group, do a focus group.  But don’t refer to it as “research” in a scientific way.

Size of Sample – there certainly is a lot of discussion about sample sizes and statistical significance and so forth in web analytics now that those folks have started to enter the more advanced worlds of test design.  Does it surprise you the same holds true for research?  Should’t, it’s just math (I can feel the stat folks shudder.  Take it easy, relax).

Without going all math on this, let’s say someone does a survey of their customers.  The survey was “e-mailed to 8,000 customers” and they get 100 responses to the survey.   I don’t need to calculate anything to understand the sample is probably not representative of the whole, especially given the methodology of “e-mailed our customers”.  Not that a sample of 100 on 8000 is bad, but the way it was sourced is questionable.

What you want to see is something more like “we took a random sample of our customers and 100 interviews were conducted”.  It’s the math thing again.  Responders, by definition, are a biased sample, probably more of a focus group.  This statement is not always true, but is true often enough that you want to verify the responders are representative.  Again, check the research documentation.

OK Jim, so how can political surveys be accurate when they only use 300 or so folks to represent millions of households?  The answer is simple.  They don’t email a bunch of customers or pop-up surveys on a web site.  They design and execute their research according to established scientific principles.  Stated another way, they know exactly and specifically who they are talking to.  That’s because they want the research to be precise and predictive.

How do you know when a survey has been designed and executed properly?  Typically, a confidence interval is stated, as in “results have margin of error +- 5%”.  This generally means you can trust the design and execution of the survey because you can’t get this information without a truly scientific design (Note to self, watch for “fake confidence level info” to be included with future “research for press release” reporting).

More rules for interpreting research…

Marketing / Technology Interface

Analytical Culture

I’m a marketing person that in one way or another has been tangled up with the technical / engineering world all of my professional life.  Cable Television, TV Shopping, Wireless, Internet.  I have always been dealing with brand new business models having no historical reference, while swimming in data to make sense of, and dealing with engineering folks as the people who “make things happen”.

I have also been really fortunate to work with many Ph.D. level statisticians who had the patience to answer all my questions about higher level modeling and explain things to me in a language I could understand.

Because of this history, I’ve been a long-time student of the “intersection” between  Marketing and Technology.  I’ve in effect become a “translator”in many ways – taking ideas from each side and converting them into the language of the other side.  Distilling the complexity of Technology down to the “actionable” for Marketing, while converting the gray world of Marketing into the White / Black – On / Off world for Technology.

With no offense to either side, to generate some kind of tangible progress, sometimes you just have to strip out all the crap from both sides to get to the core value proposition of working together.  You have to start somewhere.  Then you can build out from there.

And so I try with posts like Will Work for Data to define this intersection for others, to help both sides understand each other, and it’s tough, especially with an unknown audience varying widely in their knowledge of either side.  I try to create a “middle” both sides can understand.

Marketing folks are in the middle of a giant struggle right now with the whole accountability thing.  But i’s not so much accountability itself, because many of the best marketers have always been accountable in one way or another.  No, it’s the granularity of the accountability that is the issue; the movement from accountability defined at the “impression” and “audience” level to accountability at the “action” and “individual” level.

Here’s the challenge for Marketers: the data is different.  Impression and audience are defined by demographics but response and individual are defined by behavior.

Perhaps this will “translate” poorly, but the Technology parallel would be folks who have built a skill set around a certain programming language and then are told that language is now obsolete.  This is extremely disruptive when you have spent 20 years understanding your craft from a particular perspective.

So here’s what we need to do to make this work.  We have to find common ground.  This will mean being a little”less scientific” on the Technical side and a little “more specific” on the Marketing side.  And we work down through all this to the core.

This is the same struggle web analytics folks deal with every day, but due to the early work of many writing on this topic, the web analysts were always urged to connect analysis to business outcome.  Many are getting pretty good at it; they don’t suffer the “too much science” problems their peers in marketing research seem to run up against.

But web analytics is just a microcosm of the whole Analytical Enterprise, which may or may not be (background info this link) Competing on Analytics at this time, but is probably headed in this direction.

I submit it’s a bit early to teach most Marketing folks about statistical significance, about what types of data sets CHAID works best with, the difference between Nearest Neighbor and Clustering models, and so forth.  We can always get there after we reach the core understanding.

Right now, what we need to do is figure out how to get to the core. 

I think where I might take this is to propose some fundamental rules of understanding and see if we get both Marketers and Analysts to understand and agree on them.

You up for that?