Fear and Loathing in WA
You may recall I wrote last year about the explicit or implicit pressure put on Analysts to “torture the data” into analysis with a favorable outcome. In a piece called Analyze, Not Justify, I described how by my count, about 50% or so of the analysts in a large conference room admitted to receiving this kind of pressure at one time or another.
Since then, I have been on somewhat of a personal mission to try to unearth more about this situation. And it seems like the problem is getting worse, not better.
I have a theory about why this situation might be worsening.
Companies that were early to adopt web analytics were likely to already have a proper analytical culture. You can’t put pressure on an analyst to torture data in a company with this kind of culture – the analyst simply will not sit still for it. The incident will be reported to senior management, and the source of “pressure” fired. That’s all there is to it.
However, what we could be seeing now is this: as #measure adoption expands, we find the tools in more companies lacking a proper analytical culture, so the incidents of pressure to torture begin to expand. And not just pressure to torture, but pressure to conceal, as I heard from several web analysts recently.
One bright young analyst went “beyond the call of duty” on his analytical project. The analyst gathered relevant data not just from the WA tool, but from Finance, Customer Service – all around the company. The report painted a detailed picture of cost to acquire customers through various methods and campaigns, and was presented to the head of Marketing – also the analyst’s boss.
The analyst was told under no circumstances was this report to ever be produced again. Further, the analyst was told to destroy any “evidence” this project / report ever existed. And finally, the analyst would now be required to send all analysis through the boss first before anybody else sees it.
That’s shameful behavior for an exec. And apparently, this kind of thing is happening more and more often. I’ve heard plenty of “if we want your opinion, we’ll ask for it” stories, but this is the first time I’ve heard so many stories about concealing results.
Here’s a scary thought: what if the stories about web analytics not driving business value are primarily concealment stories? What if the tool / analysts actually did provide value, which was then hidden from Senior Management?
My concern about this issue is wider than screwed up company culture and management. What I’m more concerned about is screwed up people, analysts who may come to think this kind of behavior is normal and just part of being an analyst.
This matters because as this new generation of analysts moves to other companies and throughout the ecosystem, these pressure to torture situations could become “accepted” and even spread as “part of the game”.
It is never, ever OK to manipulate or hide the results of an analysis. It’s not part of the job. The role of an analyst is to analyze, not justify or conceal bad news.
Now, I realize some folks are thinking, “Yea, that’s great Jim, I’ll just get myself fired by being an analytical hero”.
I’m not saying you should respond to data torture pressure by falling on your analytical sword. What I am saying is you – and management – need to know this kind of pressure from a superior is shameful, not a “normal” part of being an analyst. And as soon as you can, you should get a job somewhere people respect your professional opinions. Don’t have to agree; but must respect.
Like the company you work for? Ask a buddy in Finance if they could use a web analyst. Pretty sure Finance would be interested in fully-loaded cost to acquire new customers by source!
What really troubles me about this situation is it’s rarely ever talked about, so could be worse than people might think. At the very least, Senior Management should know about the potential for this to happen and lay down some rules. Perhaps even seek some cultural guidance on this topic (here’s a start – Fear of Analytics).
So, I want to put this message out there, perhaps create a resource for people who are looking for information on this topic. It would be great to have examples so managers can understand and be on the lookout for these situations. Plus, I’m sure there are some terrific stories out there about either giving in to the torture pressure or resisting it!
What about you? Were you ever pressured to torture the data? What happened? Did you comply? How did things come out? Tell us with a Comment. Feel free to post anonymously, leave out company names.
Tags: Analytical Culture
Social comments and analytics for this post…
This post was mentioned on Twitter by diogenespassos: RT @minethatdata: Must read #measure from @jimnovo. It happens to my consulting work, it happened several times in my corporate life. http://bit.ly/c78PQe…
Jim,
Great article and really rings home to several stories I have heard in this part of the industry that I now work in (TagMan) and those on my previous life Agency side.
The most pertinent that I would love to share is that of the analyst who KNEW that deploying your analytics tags at the top of a page (we all know this now) shows a beneficial X% difference in reported traffic/results – than if you put at bottom of page (we have our own latency based study on TagMan.com).
He presented these details to the management team of his publicly listed company who promptly told him to leave it be; as the difference in reported traffic to actual results made their public-facing business reports look all the better; therefore an inflated value rather than actual.
Hi Chris – that’s a great story, thanks!
It’s a bit different than I was thinking of – in your case, at least management was presented with facts and *they* chose to ignore them. I’m actually OK with that – it’s management’s show, and the analyst retains integrity.
Much more concerned about Sr Management *never hearing* the real story because of pressure to torture / conceal data from folks below them.
Hi, Jim – been a fan of your work for years – glad someone is finally willing to expose this problem.
Briefly, I was engaged to bring accountability and optimization methodology to a very large organization’s branded internet marketing effort. In a nutshell, after facing many months of passive/aggressive behavior, I was politely told by the sponsoring, internal service agency NOT to advocate ROI as the final determinant of campaign effectiveness as it was considered “unpopular” to be that accountable.
This was in direct contradiction to the brand manager’s public requests
on a spend of $5mm annually.
Silly me, in delivering what the ultimate business client needed, the internal service agency felt their control was challenged and as a result my contract with them was terminated.
I guess developing a clear ROI business case was not in the best interests of those who have historically survived by being unaccountable, but politically expedient.
STM, Love your “Name”, thanks for the comment. Others, feel free to comment as “STM”…
Imagine, “unpopular to be that accountable”. Of course, this speaks to the larger accountability movement in Marketing, which gains a bit of steam every day, fortunately…
I often find that “forcing a hypothesis” can help with these kinds of situations. If not ROI, then what? Why are we doing this, what is the benefit? There must be *some* benefit to a spend of $5 million, right? This approach often creates some mild tension that can result in a stake in the ground. “The benefit of the campaign is web site visits”.
OK, if it’s “visits”, how many? 500,000? OK. Does a 1 page, 2 second visit count? No? How about only visits with at least 2 pages? And so forth.
If ROI can’t be the game, and you can get people to put a stake in the sand for some benefit, then at least you have something to analyze against and try to improve.
Thanks for the story!
Great post, Jim.
Few weeks ago, I solved that problem for myself: After four long (too long) years of hearing exactly what you wrote: “If we want your opinion…” and concealing results from our clients, I handed in my resignation.
Not that move was sudden. I protested as much as I could over the years (and sold my silence too many times, not something that I’m proud of). The usual excuse was: “That’s what gives you your paycheck”. There were things that I explicitly didn’t do, such as tampering with control groups, but I did conceal results. Despite my (very loud) protest, nothing changed, even when clients left us for the competition. Eventualy, I couldn’t handle it any longer.
AFD, thanks for the comment, and I admire your honesty. “Selling Your Silence” is a fantastic term that’s more to the point than the “torture” riff, and perhaps I’ll use that going forward instead (if it’s OK with you!).
I think the important point is you knew what was going on, and dealt with it in your own way; no shame there! Overt “paycheck” threats are a particularly horrible practice that cell phone recording might put away…
People need to understand that being an analyst by definition brings a certain amount of conflict, and *especially so* if you’re good at it. Deal with it as you must for where you are in your career and responsibilities, just know that selling your silence is not “part of the game”.
And, there are companies out there who respect the role of the analyst, though ad agencies seem to have a bit of a problem with it, according to response received so far. That’s not surprising, is it?
Any agencies out there want to hire an honest analyst like AFD?
Thanks for your concern, Jim
It seems that there are some positions for experienced analysts, not only in the WA area. I got some feedbacks and hope to find a new job very soon.
Jim,
Another great post. It is crazy to see all of the comments and realize that we are not alone in our struggles. This is always comforting to me. Thank you for providing a place that we can voice our concerns and know that somewhere, someone else is in the same boat.
Jim,
This is a great discussion you’ve started–sorry if I’m a little late on the draw. First, I’m a huge fan of your blog, your book, and your work, but I must admit to being surprised at, well, your surprise.
Maybe you haven’t worked for very many large companies?
This sort of thing is positively endemic, and not just in web analytics, but in all aspects of Corporate America, but especially management consulting and Wall Street. At my last job I worked for a very sharp former McKinsey consultant and Harvard MBA, who was new to web analytics, but had many years of data analysis and presentation experience. He was surprised (and delighted) to find that I would actually look at the data first, develop hypotheses, test them, and then prepare presentations with the insights and recommendations.
Why was he surprised? Because he’d never known anyone who actually had done this before. In his experience, the process involved an “understanding” with the client…He’d start with the decision that the client wanted to make, and then “torture” the data to find evidence to justify the decision so that the client could sell this decision internally to employees and externally to the media (and to Wall Street of course). Spin 101.
Speaking of Wall Street, and Wall Street analysts, they (for the most part) rarely do what you would call “real” analysis; instead the analysis is the marketing.
As much as it pains me to say it, in the real world, VERY few large organizations have a true analytical culture, and the higher you go up the org chart (especially if it’s a publicly traded company), IME the less likely any real analysis is done and the more everything becomes a matter of appearances rather than substance.
So I think you’re really touching on a subject that is much bigger than web analytics. It’s about organizational behavior, power, control, and spin.
NotMyRealName, I’m shocked that you’re shocked that I’m shocked.
I have worked at some big companies, both as an employee (HSN is a $2 billion operation) and a consultant (Verizon is kinda big) and I have seen the analytical culture work and not work, sometimes in the same company!
HSN was basically a data fist fight – EVERYBODY had analysis so disagreements were often decided by methodology; it was truly a data-driven culture. This is what happens when a minute by minute sales and profits dashboard is on every computer Director level and above, I guess…
What really bothers me in terms of web analytics is this: people will sit around and say they are “not getting any value out of their web analytics investment”. And that could be true = weak analytical talent.
Or it could be that they’re not getting the real story out of their WA investment because everybody is so busy torturing the data to cover their arses that no action is ever taken.
And I just wonder, what % is the former, what % is the latter?
I guess we can only hope that Darwin will take over and the companies who are truly “Competing on Analytics” will survive and the data torturers will not!