Monthly Archives: February 2007

I’m at Training 2007 (the Conference)

Analytics Education

Yea, I know, kind of weird.  What the heck is a Marketing / Web Analytics guy doing at this event?

The Training Conference and Expo is the largest conference of training professionals in the US.  It’s the first conference I have been to in 10 years that I’m not speaking at.  Probably the first conference I have been to in 20 years where just about everybody knows more about the topic than I do. 

And I have to tell you, that’s incredibly refreshing. 

I’m thinking I have to do this more often!  After all, what exactly is the point of going to conferences on material you already have deep knowledge of?  Unless it is to present, of course…

I’m here on behalf of the Web Analytics Association scouting out vendors to administer the Certification test we are developing for web analysts, and to learn everything I can about best practices in Certification.  One of the challenges is we are looking to certify folks not on “software” related issues like implementation / set-up (the vendors do a fine job here) but on the business side, where the issues are often not as quantifiable as they are in software land.  So we need a vendor that can work with us on a more flexible testing methodology than many are used to.  If you have any suggestions / advice on certification test vendors, let me know.  There are 9 vendors here.

Here are some interesting things I have learned so far:

1.  Virtually none of these Training / HR folks have ever heard of web analytics before.  They have no idea what the heck I am talking about, or that web analytics people even exist from an HR perspective.  The typical response is “we could have used somebody like that when we were setting up our Intranet … what is their typical job title and who do they report to?”

 2.  The primary model used in training course development is called ADDIE.  It stands for:

Analyze
Design
Develop
Implement
Evaluate

which is a formal sequence of tasks where “Evaluate” has an arrow looping back up to the top pointing to Analyze, meaning you repeat the sequence and there is a continuous improvement process.  Hmm, that sounds kind of familiar, where have I seen this before?  Perhaps filed under Best Practices for web site development?

3.  Lots of the communication and behavioral models used in Marketing are used in Training - Training is in many ways a specialized kind of Marketing.  I initially thought I was dead wrong about this but when I put forth the idea, nobody threw me out of the room or called me a Newbie.  So I think there is something worth exploring about this parallel, especially since e-Learning delivered through web interfaces is a big deal to these folks.

More to come as the event unfolds…

More Trouble for Unique Visitors

Web Analytics

I’m minding my own business and McAfee wants to “Update”.  I think this is a simple update of the virus database and even though I am very busy doing something else, I go for the update.  So of course, without warning, I am treated to a monster update of the entire McAfee program, complete with all kinds of FUD links that lead to very poorly executed landing pages.  Terrible customer experience, and that’s what I was going to post about.  But since everybody probably sees the same thing all the time, I think the following sequence will be much more interesting.

Being a web analytics freak, about 30 minutes after the install, I checked out the way the new McAfee program handles cookies.  You guessed it:

(click on any of these images for bigger pic)

“Scan and remove tracking cookies” is automatically activated on install.  Then I go into the “Quarantine” section and here is what I find:

All my “tracking cookies” have been Quarantined.  Certainly looks like all the major ad-serving networks are represented, and what looks to be a bunch of Overture conversion cookies.

This cookie crunching doesn’t mean much to me because I don’t build any very important (KPI level) metrics using “Unique Visitors” as a base.  For one thing, I was doing web analytics before cookies were pervasive and I’m comfortable using “Visits” or “Sessions” as a base (Sales per Visit, for example, as opposed to Sales per Unique Visitor).  The other reason is that you simply cannot get an accurate Unique Visitor count, meaning there’s a lot of “noise” in the number.  I don’t like basing key performance metrics on a noisy base number, it’s asking for trouble.  And it appears this cookie situation will be getting worse over time – worse than it already is with all the anti-spyware scrubbing of cookies, the firewall problems, and so forth.

Yet I know a lot of people base everything they do on Unique Visitors because it “makes more sense to management” and it’s “more logical” and so forth.  Fine.  Here’s what is going to happen.  The cookie block / erase / quarantine problem is going to artificially increase the number of Unique Visitors you are getting to the site.  You’re not getting more, it will just look like you are due to loss of tracking at the Unique level.  This means Sales per Unique Visitor, for example, will start falling over time even though in reality, based on actual Unique Visitors (which you can’t measure) it may be staying the same or rising.

My advice to you is to start shadow tracking now using Visits or Sessions as the base in your most important metrics, the ones you are on the hook for.  You don’t have to show them to anybody, just keep track of them in Excel or something and note the trends.  Then when you start seeing your Unique Visitor based metrics collapsing on you, you can whip out the Visit / Session based metrics and say, “See!  See!  It’s really not happening!  We’re doing much better than you think!”

Makes more sense to management, indeed.  Until you try to explain why Management should now believe your Visit-based metrics instead of the Unique Visitor based metrics.  Good luck on that one.

(Note to Ron: This subject makes me very Cranky, could you tell?)

***** The Medium is the Metric for Online Ads

Analytical Culture

Article with the title above, published here, is apparently creating quite a stir in the advertising media community, particularly among the “brand” folks.  One of the core points is that all media will become measurable and thus “accountable” in terms of the effect ads placed in the media have.  While I’m not sure that’s going to happen in my lifetime, the initial thrust is that online media better get their crap together in the measurement area and define some standards, because the new “Agency of Record” is going to be an analytics shop that measures, in a centralized way, the effectiveness of all advertising a client is running.  The unspoken implication here is this agency would essentially have the power to fill or kill any campaign based on performance.  Neat idea.  Two comments:

1.  I was selling cable television ads in the mid 80’s when Nielsen, despite intense pressure from the broadcast networks, started metering cable homes “in the box” (wired into the set top controller).  When the first hard numbers came out, they absolutely blew away all the estimates of the cable viewing audience.  Turns out a lot of cable viewing was not captured in the paper diaries, and the meters picked it up.  Go figure.  You mean what people report to you in a survey doesn’t reflect their actual behavior?  C’mon, that can’t be true (being sarcastic for those who don’t know me).  Anyway, network cable advertising absolutely exploded after this, and all this money fueled better programming.  That’s when “Big 3” share really started to tank.

In other words, we have seen this movie before.  Money follows accuracy.  Instead of resisting this idea, these brand folks ought to embrace it and hang on – it’s going to be a wild ride.

2.  The idea of a central agency being the “Master Record Keeper” is an absolute must, since if each agency runs it’s own success metrics, each agency will claim success that really belongs to another agency.  You need to have a source of the “one truth”.  I have argued this same point many times with companies that have analysts spread out into each of the silos.  While it is possible this could work, you would need an iron fist to enforce consistency and remove the tendency of an analyst to paint a better picture of the silo his boss runs. 

Just trying to make sure every silo is being honest would take a huge amount of work – why not just centralize it in the first place, and do the work once?  If all the analysts report to a CAO who basically is a 3rd party with no axe to grind, then the CEO is going to get the straight picture – including all the cross-silo effects, which is usually where all the ROI is hiding.  For example, you are not going to get an analysis that includes the “true cost” of a marketing campaign that causes all kinds of problems in customer service from an analyst in the marketing department, it’s just not going to happen.  You need a 3rd party view to get to the Root Cause and start fixing broken processes that affect the customer experience and waste a ton of money.  There are very positive benefits to having a group of analysts who each are experts on a single piece of the company under one roof, interacting and discussing business issues.  That’s how you get breakthrough thinking, how you fix the broken cross-silo processes that drive customers crazy.

It’s great that we are all becoming more accountable, but let’s get down to the meat of the matter and kick analytics up to the C level and out of the silos.  How long will it take before the CEO finds out the silo analysts are “torturing the numbers”?  Do you want to be there when it happens?  It’s not pretty, let me tell ya.  I’ve seen it.

In case this doesn’t make any sense to you, here is an example.  How many customers does your company have?  Ask 5 people, you will get at least 3 different answers (if you get any answers at all) and they are all probably wrong.  The proper answer is “that depends on how you define a customer”.  Now, picture each silo with their own set of KPI’s, many based on the “number of customers” in some way and you start to understand what I am talking about.

Like I said, not pretty.  A full-on, CEO’s beating forehead vein kind of thing, a “We’ve been telling Wall Street we’re running this company based on analytics and now you tell me that we can’t even agree on how many customers we have?” kind of meltdown.

Maybe you should start thinking about centralizing your analysts now?  Or at least talking about it?