Reporting versus Analysis: The “Actionable” Debate

Gary Angel and Eric Peterson have been having a great exchange surrounding the definition of KPI’s, and more specifically, the requirement that they be actionable.   Gary started out with the position the “criteria of actionability is unsound in almost every way” but I think both he and Eric have resolved in the middle somewhere – it’s really about context.  Gary is right, to take any metric “naked” at face value without surrounding context is simply not good analytical practice.  But I would argue (and I think Eric agrees) that to build a KPI in the first place, you must already have the required context, or you don’t have a KPI.  So that leaves us with “how you define a KPI” as (I think) the final resting point, and there really isn’t anywhere to go after that.  Your comments on my analysis welcome.

However, I think the ideas Gary has exposed run deeper than just the KPI discussion.  The situation Gary is addressing – making sure people really understand that every metric requires business context to be functional - requires attention because web analytics is a very fast growing field with a lot of brand new people in it who may have not been exposed to proper analytical training. Or, not challenged to do any real analysis by weak managers.

These new people frequently don’t understand the difference between Reporting and Analysis.  A “Reporting” mentality (provide data) leads to the improper use of analytical ideas like KPI.  Analysis (provide insight) would automatically take into account a lot of other factors, as Gary has suggested.  Knowing all those factors (because you are doing real analysis), you can certainly take movements in a KPI as actionable.  As Eric says, that “action” is often a more focused analysis of some kind.  KPI’s are really just “tripwires” that alert you to a problem or opportunity that requires further analysis.

My concern (and in the end, I think Gary’s) is that often the Reporting mentality is Robotic and that the reaction taken to change in a KPI might be equally Robotic if you don’t have the proper context.  What often happens in Pay-per-Click testing is a great example of this, and a lot of the multivariate stuff people are now addicted to is an extreme example. 

You can look at conversion rates, make changes to landing pages, and try to optimize the “Scenario”.  This is Reporting, not Analysis.  Can you provide insight into why the changes you made worked?  For example, can you explain the improvement in terms of Psychology or Consumer Behavior?  Usability?  If so, that would be Analysis, and the answers would be applicable to a wide range of other challenges on the site.  Without knowing why the changes worked, you are left with simple Reporting that applies to only a single specific Scenario.  Nothing was really learned here.

Take this same idea to the extreme, and you get what often happens in multivariate testing.  You can certainly run a multivariate test on 5 variables at the same time, and find a “winning combination”, but this is Reporting, not Analysis – in fact, it’s black-box reporting in the extreme.  For example, how do you know that you chose the 5 most important variables to optimize?  How do you know the options you chose for each variable are the most powerful?  Isn’t it just as likely that the final optimization you achieved is suboptimal, a local maximum, as it is the solution is truly optimal? 

In other words, isn’t it possible that what you have created with the robot is better than you had, but is not even close to being the best it can be?

Dear Reader, you’re asking, why should I care about this Reporting versus Analysis issue?  Because here is what will happen without real Analysis: you are going to “hit the wall”.  One day, there will simply be nothing left you can do to improve on what you have done.  Reporting is only going to take you so far.  Frustrated, you will probably Analyze the situation and realize you have “optimized” yourself into a corner by taking something that was fundamentally broken in the first place and making it better than it was.  You can’t make it any better unless you wipe it out and start again.  That’s a huge waste of resources, right?

See CRM if you need an example of what can happen when you automate worst practices.  And they’re going to fix it 8 years later by bolting on Business Intelligence?  Um, shouldn’t the Analysis have come first?

9 thoughts on “Reporting versus Analysis: The “Actionable” Debate

  1. Jim,

    Great post. I think Eric and I did come out somewhere in-between and I agree with almost everything you wrote. Only thing – I’d be even more skeptical than you about how quickly you “hit-the-wall” with a reporting mentality. Multivariate testing is the one place you can actually go for awhile (productively) before you do hit a wall without additional analysis. But in most other ways, I see organizations hit an almost immediate dead-stop because they are focused on reports as triggers to action and never do any real analysis.

  2. Kiran, thanks for the comment.

    I’m sure this subject will be coming up more and more as WA folks gain experience. Probably ends up with a “split” in career track and some point; you have the implementation folks who are also probably Reporters and then you have the Analysts who are really driving the bus.

  3. Reporting vs. Analysis is like Data vs Information.

    A report on sales over the past 6 months doesn’t actually tell me how well my company is performing. Once I add things like market growth, product introductions, customers, competition, profitability, number of sales people available, etc. to the equation, then the sales figures (information) starts to make sense to me. Analysis is all about explaining how you got somewhere and where you – most likely – will be going to

    I wonder how many companies really do this kind of analysis

  4. Thanks for the comment Marcel. The best answer I can think of for your question on how many companies do this kind of analysis is it depends on how competitive the industry is. Online travel, for example, has some of the brightest analytical stars and the best infrastructure because without it it they will go out of business. Likewise Proctor & Gamble offline.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.