Category Archives: Web Analytics

Reporting versus Analysis: The “Actionable” Debate

Analytical Culture, Marketing / Tech Interface, Web Analytics

Gary Angel and Eric Peterson have been having a great exchange surrounding the definition of KPI’s, and more specifically, the requirement that they be actionable.   Gary started out with the position the “criteria of actionability is unsound in almost every way” but I think both he and Eric have resolved in the middle somewhere – it’s really about context.  Gary is right, to take any metric “naked” at face value without surrounding context is simply not good analytical practice.  But I would argue (and I think Eric agrees) that to build a KPI in the first place, you must already have the required context, or you don’t have a KPI.  So that leaves us with “how you define a KPI” as (I think) the final resting point, and there really isn’t anywhere to go after that.  Your comments on my analysis welcome.

However, I think the ideas Gary has exposed run deeper than just the KPI discussion.  The situation Gary is addressing – making sure people really understand that every metric requires business context to be functional – requires attention because web analytics is a very fast growing field with a lot of brand new people in it who may have not been exposed to proper analytical training. Or, not challenged to do any real analysis by weak managers.

These new people frequently don’t understand the difference between Reporting and Analysis.  A “Reporting” mentality (provide data) leads to the improper use of analytical ideas like KPI.  Analysis (provide insight) would automatically take into account a lot of other factors, as Gary has suggested.  Knowing all those factors (because you are doing real analysis), you can certainly take movements in a KPI as actionable.  As Eric says, that “action” is often a more focused analysis of some kind.  KPI’s are really just “tripwires” that alert you to a problem or opportunity that requires further analysis.

My concern (and in the end, I think Gary’s) is that often the Reporting mentality is Robotic and that the reaction taken to change in a KPI might be equally Robotic if you don’t have the proper context.  What often happens in Pay-per-Click testing is a great example of this, and a lot of the multivariate stuff people are now addicted to is an extreme example. 

You can look at conversion rates, make changes to landing pages, and try to optimize the “Scenario”.  This is Reporting, not Analysis.  Can you provide insight into why the changes you made worked?  For example, can you explain the improvement in terms of Psychology or Consumer Behavior?  Usability?  If so, that would be Analysis, and the answers would be applicable to a wide range of other challenges on the site.  Without knowing why the changes worked, you are left with simple Reporting that applies to only a single specific Scenario.  Nothing was really learned here.

Take this same idea to the extreme, and you get what often happens in multivariate testing.  You can certainly run a multivariate test on 5 variables at the same time, and find a “winning combination”, but this is Reporting, not Analysis – in fact, it’s black-box reporting in the extreme.  For example, how do you know that you chose the 5 most important variables to optimize?  How do you know the options you chose for each variable are the most powerful?  Isn’t it just as likely that the final optimization you achieved is suboptimal, a local maximum, as it is the solution is truly optimal? 

In other words, isn’t it possible that what you have created with the robot is better than you had, but is not even close to being the best it can be?

Dear Reader, you’re asking, why should I care about this Reporting versus Analysis issue?  Because here is what will happen without real Analysis: you are going to “hit the wall”.  One day, there will simply be nothing left you can do to improve on what you have done.  Reporting is only going to take you so far.  Frustrated, you will probably Analyze the situation and realize you have “optimized” yourself into a corner by taking something that was fundamentally broken in the first place and making it better than it was.  You can’t make it any better unless you wipe it out and start again.  That’s a huge waste of resources, right?

See CRM if you need an example of what can happen when you automate worst practices.  And they’re going to fix it 8 years later by bolting on Business Intelligence?  Um, shouldn’t the Analysis have come first?

Is Your PPC Incremental?

Analytical Culture, Web Analytics

Something that analytical folks talk a lot about is “the incremental”.  In customer marketing, it’s not enough to get response, what we want to know is how much of the response was above and beyond what we could be expected had the test or promotion not been done.  Typically incrementality is measured using control groups – people just like the people you are sending the campaign to who do not receive the campaign.  Then you compare the profitability of the people who received the campaign versus the profitability of those who didn’t.  The difference is the true, incremental profitability of the campaign.

Kevin over at the MineThatData blog relates that Blue Nile has decided to lower prices as opposed to chasing rising ad costs.  I don’t know enough about their business model to really comment on that action, but I do know one thing – they (and many, many other companies) are wasting Pay-Per-Click spend on non-incremental clicks.

Check out this shot.  Search for Blue Nile in Google, you get a Pay-Per-Click ad for Blue Nile, and Blue Nile has the fiirst natural listing:

Search: Blue Nile

 

 

 

 

 

 

Now someone, somewhere, back in the beginning of PPC probably had a conference speech that told everybody to buy the search for your company’s brand name, maybe for “branding” or “exposure” purposes.  But almost every web site ranks first in the natural rankings for their own brand name.  So I ask you, is there any incremental here?  What are the chances, if the PPC ad was not there, that the visitor searching “Blue Nile” would not click on the first natural listing?

What do you suppose the volume of this search is, and how much money is being wasted by people clicking on the paid ad who would have clicked on the natural link anyway?  I asked the same question back in 2003, and though the structure of the test wasn’t as pure as most hard core test & control folks would like to see, the results were so dramatic that you can’t really argue much with the methodology.  A few percentage points here or there and you still get to the same place.

In this particular test, on a high volume PPC phrase for a site that also had a top 3 natural ranking, 77% of PPC sales were non-incremental – stolen from the natural side – and would have happened anyway without the PPC link.

I will be the first to say that results are going to vary a lot by the type of site, the phrase, the target audience, the search engine, and so on.  But Golly Gee folks, has anybody else ever questioned the wisdom of “railroading” a top natural listing with a paid listing – and especially for your own brand name?  I bet a lot of analytical folks have, they just couldn’t convince the marketing folks to test it.  After all, addressing / fixing this by taking down the PPC ad means:

1.  There will probably be a few lost sales, perhaps from people who for whatever reason, always click on paid ads (newbies?)  However, profits will increase.  Some folks will not believe in this idea, it’s that whole analytical culture problem.  You have to trust the numbers, not fear them.  Unless, of course, you have been specifically told to grow sales and not profits.  Rare, but it does happen.

2.  Then there is the whole “How could you not know this was a bad thing and continue doing it for so long?” problem.   That’s more analytical culture stuff.  Failure is a learning experience in the analytical culture, not a cause for whippings and firings and demotions.  The question is not how much money was lost, the question is how much will be saved going forward.  Right?

3.  Potentially, some PPC budget will be reallocated to other, more profitable marketing activities.  This could be a problem if there is “ego spend” involved in the management of PPC.  I wouldn’t move the money out of PPC, I would just find better ways to use the existing budget.  But the potential exists for losing it for a higher and better purpose that is proven out by the numbers.  Just optimizing the system, no hard feelings, right?

PPC Marketers, if you’d like to free up budget for something else, ask your web analytics folks if they can track this test for you.  Locate a high volume search phrase that you buy PPC for ads for where your web site naturally ranks as #1 or #2 for the same phrase.  Try shutting the PPC for that phrase off.  How much click volume do you lose versus the cost of the clicks when PPC is tuned on, and what does that translate into in terms of increased profit?

And I am really not picking on Blue Nile here – search any brand name, online or off.  Go ahead.  What do you see?

Would you fail to click on the organic link if the paid link was not there?  Really?  If you think you’d miss the organic link if the PPC link wasn’t there, send me a pic of the example.

Top Exit Pages Data – Useless?

Web Analytics

Avinash says on his blog, “For the most part you should not care about this metric [top exit pages], for most websites it tends to be a hyped up metric that tells you little while, on paper, claiming to tell you a lot.” Most of the commentary on his blog seems to agree with him.

C’mon folks.  Methinks when you are as skilled as Avinash is, or you have been analyzing the same web site for a long time using advanced tools, sure, the metric can be next to useless. I don’t even look at the “Overview” stats for any of my sites anymore, because I have Custom Reports that tell me what I really want to know.  I advise clients to take the same approach.

But, when you sit down to analyze a site you have never analyzed before, there is nothing that can orient you like first looking at Top Entry and Top Exit pages. For me, it creates a visual map of the basic behavior on a new site and causes me to start asking the questions that will be the subject of further study. And for a person that is brand new to web analytics, simply seeing Top Entry and Top Exit stats for the first time has a huge impact, it gets them to the “First Why?”, if you know what I mean. It’s a very simple model that is easy to understand.

Perhaps interacting with the students in the UBC / WAA web analytics courses has made me more sensitive to this issue, but I think we might better qualify statements like this, for example, “For the most part, advanced users should not care about this metric…” because there are a lot of folks “listening” out there who might lack enough context to correctly parse a statement like this.