I think this is probably the last Learning and Teaching issue in Online Marketing (series starts here) before attempting to evaluate and summarize the challenge. I would like to receive comments from you on the Sherlock Holmes Problem.
“There are two types of minds — the mathematical and what might be called the intuitive. The former arrives at its views slowly, but they are firm and rigid; the latter is endowed with greater flexibility and applies itself simultaneously to the dive.”
In his post How the Skills of a Night Auditor Translate into Web Analytics, Christopher Berry explores a notion we have wrestled with a lot while developing the WAA’s Certified Web Analyst Test – can you teach someone to be curious in a “business analytics” way? Or are people just born with / socialized into this skill set? How do you measure and test someone for “analytical curiosity”?
We have referred to these issues internally on the Education Committee as the “Sherlock Holmes” problem. The issue is not the ability to read and interpret reports, or write up findings, or anything like that. It’s the ability to see coincidences or oddities in the data, to conceptualize linkage or relationships others don’t see, to follow the data trail (or blaze it) right down to Root Cause.
From what I have seen over the years, in both web analytics and traditional BI, there are many technical masters of the analytical tools. Relatively few of these folks, however, also have the “Sherlock Holmes ability”. They can answer any question you have, but they don’t come up with the questions. That’s the Holmes side – coming up with the breakthrough questions. And figuring out how to answer them. The analysis, well, that’s just the mechanics.
Our friend Pascal above speaks of the “mathematical” and “intuitive” minds. I often wonder if there is something about learning “Programming” and “Marketing” that creates a rigid way of learning on either side that is mutually exclusive with the other. The intersection or “cross” of these two mindsets, the corpus callosum if you will – is seemingly rare. Can you train / learn to cross your brains? I have referred to this idea as “being a little “less scientific” on the Technical side and a little “more specific” on the Marketing side“.
Having spent a good deal of time in the hospitality biz myself, I can tell you there are night auditors who just “run the reports” and night auditors who deliver great insights; these are the Holmes. A common “fault” you hear about these folks is “they ask too many questions”.
My wife says I ask too many questions. Why do you want to know?
I dunno, I just want to understand it.
But why do you need to understand it?
I dunno, I just do. I like understanding things.
Can being a night auditor teach you to become a great analyst, if you work at it hard enough? Or were you somehow born or socialized into being a great analyst, and night auditor is just one of those analyst jobs? There are three related “Sherlock Holmes” Learning issues in my mind: Pattern Recognition, Empathy, and Hypothesis.
Pattern Recognition – the “math” brain?
I have always described my particular skill in analytics as “I see patterns” (I know, cue the music). I’m not sure where this skill comes from, but it’s been a skill for as long as I can remember, from taking music lessons through working on case studies in school and on into my business career. Perhaps more importantly, I know how to translate these patterns into Marketing programs that generate increased profits (but that’s the “other brain”, see below).
Much of what Christopher talks about in his post is pattern recognition, or often more importantly, the deviation from the patterns you expect to see. For example, (Hypothesis) closing the kitchen at 11 PM will increase total liquor sales across the dining room and bar. Once you analyze and prove this, it’s easy for others to analyze and follow (and post on blogs and have conferences about!)
But how about the ability to even conceive this question in the first place? Not to mention running through the possible outcomes – what if they drink less $100 a bottle wine in the dining room and instead have $5 drinks in the bar? How many $5 drinks would they have to have? Is my projection logical?
I know, depends on the audience…
But seriously, how do you teach people Pattern Recognition, or to Recognize Lack of Pattern? How do you test people for this capability? Rorschach Tests?
Empathy – the “marketing brain”
I’ve seen a lot of material on analytical thought – right brain versus left brain mix, deductive versus inductive thinker, etc. Sometimes I think analytical ability has a lot to do with Empathy – can you understand and visualize the players, think about what they would do. How many “boozers” are there in the dining room on a Saturday night, and what would they do if we closed the kitchen at 11 PM?
Perhaps this pull towards Empathy as a key component of being Sherlock Holmes is my Marketing background speaking, because it’s very common in Marketing to have to “step into the shoes” of the customer or audience. Again, I can tell you from experience, it is difficult for many analysts to “Pretend you are a woman over 50”. Even more difficult – and often more critical – is to then answer the question “Given you are a woman over 50, what would you do or think?” The concept of Personas is meant to address some of this problem, make the challenge more tangible for people. But you still have the “what would the Persona do or think” issue.
How do you Teach or Learn this skill? I think there is a concrete answer here – you simply have to read a lot of valid research (not this kind), and then take the time to think about it, connect it to experience. I do see the tremendous amount of twisted and corrupt research that gets passed around the blogosphere as “truth” being a severe Teaching / Learning challenge though. No wonder people are confused and always searching for “the truth”.
Hypothesis – the brains meet
Once you know using Landing Pages is a good idea, it’s not very hard to measure and report on that idea, replicate and copy it, spread it through conferences and blogs. It’s an important idea, so important in fact we are still talking about it 10 years later.
But somebody had to invent the concept of a Landing Pages in the first place. And when that idea came out of seemingly thin air, also devise a way to prove it was a good idea. Hypothesis and Test. It seems to me Online Marketing / Web Analytics is absolutely chock full of Tests, many of them without a Hypothesis. Instead, we have a lot of brute force Testing, and then a winner is chosen.
This could be why there is a glacially slow learning curve; in fact, all this testing is “one-off” activity and people really are not Learning the fundamentals, not connecting the dots, as it were. So these folks just keep Repeating the Past. Testing without a Hypothesis is a very robotic way to look at Online Marketing and Analysis, and at the very least wastes a lot of time. This brain “Arrives at its views slowly, but they are firm and rigid”, as Pascal would say.
Do people really have to repeat the “Landing Page” test with every new piece of online media? I guess you would have to, if you didn’t have the Hypothesis “message continuity in a sequence of online advertising pieces will result in a higher response rate”. I can tell you it does. I can tell you it always has offline as well. It’s just a fundamental rule, especially in direct marketing, for example:
Outer Envelope > Teaser > Open Envelope > Johnson Box > P.S. > Letter > Reply Device
The better integrated these messages are – the stronger the “scent” – the higher the response. There is not a need to keep testing this idea. Think about it – does it make any sense to you, is it possible that a random series of disconnected messages would get a higher response rate? Really?
Lack of Hypothesis for each test may be the most crippling Learning problem out there, and perhaps the most difficult to crack because it requires both brains to be in gear at the same time. In other words, creating a breakthrough Hypothesis probably requires both Pattern Recognition and Empathy.
In case the concept of Hypothesis is not quite clear, I don’t view an “educated guess” as a Hypothesis; you need to provide concrete reasons for the Hypothesis, fully explaining why the outcome should occur. The good news underlying the Hypothesis approach is this: by being proved wrong or right every time – instead of just waiting for outcome – folks will start to Learn. Just shrugging your shoulders and accepting the outcome of a test doesn’t create Learning.
Write it down, what do you think the result of the test will be and why? At some point, people will begin to say to themselves, “Hey, this test looks like these other ones. My Hypothesis is…”
And more often that not, they will be correct. Sometimes they will be spectacularly wrong, and that’s OK, because Failure is a Learning Experience.
So that’s the Sherlock Holmes Problem.
Comments on any of the above? Are great Analysts born, socialized, taught? Have I completely missed the core issue, and something else – lack of proper analytical culture, for example – is really the root cause of a slow learning curve? HIPPOS don’t know how to form a Hypothesis, and resent one being formed for them?
Anyone know of resources (preferably focused on education) dissecting these issues / providing learning solutions?Follow:
3 thoughts on “Sherlock Holmes Problem”
A mathematical mind is naturally curious as maths requires ingenuity to solve its often abstract problems, creative minds might not have the discipline to understand the data well enough. But pattern recognition gets a boost from a creative mind that can see the ‘gabs between the branches’ and come up with unexpected ideas/explanations. Someone else said a varied background tends to create better analysts, maybe because they have more varied experiences and can ‘empathise’ better?
Truly original thoughts are very rare, most of the time a great analyst is putting together the various bits of knowledge they have and coming up with reasonable hypothesis. This is where growing up with Internet means you understand it better – understand need for consumer focus and how relationship marketing can get you there.
As for a lack of proper analytical culture I really believe this is critical and where most companies fail. Nowhere I have worked actually uses analytics to drive the thinking – maybe because it’s a longer-term strategy and the usual corporate greed gets in the way.
I’d go with extracts of Chico’s response: Great Analysts are grown via experience. Wide ranging experience. The experience thus gives the background to have a better appreciation of the questions to go hunting/asking for.
I’d mildly disagree that “Programming” teaches a rigid mindset tho. Quite the opposite. The actual code is rigid – syntax et al; But the solutions are anything but rigid and more akin to an art form.
A favourite example is the wonderful “Story of Mel”:
I have often felt that programming is an art form,
whose real value can only be appreciated
by another versed in the same arcane art;
there are lovely gems and brilliant coups
hidden from human view and admiration, sometimes forever,
by the very nature of the process.”
PS Apologies for the delayed response – been in London the past week and ultra time poor
Thanks for the Comments!
One of the most difficult parts of figuring out this Repeating the Past problem – and then addressing Learning and Teaching – is trying to assess how much of the “blame” belongs on which side of the brain – math or creative. Or if you prefer, how much blame the “Technology” folks deserve versus the “Marketing” folks.
Know what I mean?
I’m leaning towards the Math / Technology side (Wrong Model), with a hard assist from the Creative / Marketing side (Dumb Money), as I indicate in the next post.
Reason? Advertising folks can only buy what’s available for inventory, and the quality of that inventory is controlled by the Tech side. Failure to innovate in this inventory area – by just slapping up Display inventory as the solution in development cycle after cycle – is the “Root Cause” of Online not getting it’s share of Attention and Budget from the C-Level.
Not that the Marketing side is blameless at all – what is needed is an Intervention by Marketing on the Tech / VC side, or the Tech side to say, OK Marketing, what do you want?
It’s the old “requirements” problem, if you will.
Either way, Marketing could say, “Here is the model that makes sense; here is the model we will buy, here is what you need to supply in order to really compete for budget with offline media”. If Marketing knows these answers, that is.
Otherwise, seems to me we are destined to keep developing great new Technology that never delivers on the Marketing promise.
Who wants to write the Requirements doc with me?