Gary Angel and Eric Peterson have been having a great exchange surrounding the definition of KPI’s, and more specifically, the requirement that they be actionable.   Gary started out with the position the “criteria of actionability is unsound in almost every way” but I think both he and Eric have resolved in the middle somewhere – it’s really about context.  Gary is right, to take any metric “naked” at face value without surrounding context is simply not good analytical practice.  But I would argue (and I think Eric agrees) that to build a KPI in the first place, you must already have the required context, or you don’t have a KPI.  So that leaves us with “how you define a KPI” as (I think) the final resting point, and there really isn’t anywhere to go after that.  Your comments on my analysis welcome.

However, I think the ideas Gary has exposed run deeper than just the KPI discussion.  The situation Gary is addressing – making sure people really understand that every metric requires business context to be functional – requires attention because web analytics is a very fast growing field with a lot of brand new people in it who may have not been exposed to proper analytical training. Or, not challenged to do any real analysis by weak managers.

These new people frequently don’t understand the difference between Reporting and Analysis.  A “Reporting” mentality (provide data) leads to the improper use of analytical ideas like KPI.  Analysis (provide insight) would automatically take into account a lot of other factors, as Gary has suggested.  Knowing all those factors (because you are doing real analysis), you can certainly take movements in a KPI as actionable.  As Eric says, that “action” is often a more focused analysis of some kind.  KPI’s are really just “tripwires” that alert you to a problem or opportunity that requires further analysis.

My concern (and in the end, I think Gary’s) is that often the Reporting mentality is Robotic and that the reaction taken to change in a KPI might be equally Robotic if you don’t have the proper context.  What often happens in Pay-per-Click testing is a great example of this, and a lot of the multivariate stuff people are now addicted to is an extreme example. 

You can look at conversion rates, make changes to landing pages, and try to optimize the “Scenario”.  This is Reporting, not Analysis.  Can you provide insight into why the changes you made worked?  For example, can you explain the improvement in terms of Psychology or Consumer Behavior?  Usability?  If so, that would be Analysis, and the answers would be applicable to a wide range of other challenges on the site.  Without knowing why the changes worked, you are left with simple Reporting that applies to only a single specific Scenario.  Nothing was really learned here.

Take this same idea to the extreme, and you get what often happens in multivariate testing.  You can certainly run a multivariate test on 5 variables at the same time, and find a “winning combination”, but this is Reporting, not Analysis – in fact, it’s black-box reporting in the extreme.  For example, how do you know that you chose the 5 most important variables to optimize?  How do you know the options you chose for each variable are the most powerful?  Isn’t it just as likely that the final optimization you achieved is suboptimal, a local maximum, as it is the solution is truly optimal? 

In other words, isn’t it possible that what you have created with the robot is better than you had, but is not even close to being the best it can be?

Dear Reader, you’re asking, why should I care about this Reporting versus Analysis issue?  Because here is what will happen without real Analysis: you are going to “hit the wall”.  One day, there will simply be nothing left you can do to improve on what you have done.  Reporting is only going to take you so far.  Frustrated, you will probably Analyze the situation and realize you have “optimized” yourself into a corner by taking something that was fundamentally broken in the first place and making it better than it was.  You can’t make it any better unless you wipe it out and start again.  That’s a huge waste of resources, right?

See CRM if you need an example of what can happen when you automate worst practices.  And they’re going to fix it 8 years later by bolting on Business Intelligence?  Um, shouldn’t the Analysis have come first?