Some of you might know that I produced the Web Analytics Association’s continuing education courses on Web Analytics (with Co-Chair Raquel Collins and a ton of great contributors).  One part of this project I am very proud of is the 4th Course, Creating and Managing the Analytical Business Culture.  This course was a real “stretch” goal because initially, the Education Committee thought there simply would not be enough material to create an entire course on this subject.  But we did it, and more importantly, the students really love it, because it addresses the “politics of analytics” they face every day.

One of the exercises in this 4th course is called The Analyst’s Enigma.  It always generates a lot of intense discussion and without giving away too much of the actual exercise, I thought you might like to either comment o how you would handle this situation or share your own similar war stories and how you handled them.

By the way, it’s based on an actual situation you may well have experience with (or will in the future) as web analytics folks spread their wings and bring their business optimization skills to the rest of the company. Here goes:

The Analyst’s Enigma

You are the manager of web analytics at a large public company.  The “web stuff” in the company started out in IT, so due to legacy reasons you are part of IT, but your internal client is really the marketing department, who is now in charge of the web site.  About 95% of your time and effort is spent working with marketing to optimize the web site, a project that has been very successful.

Over time, your success with the web site optimization process and as an analyst has been recognized in the company, and you occasionally get requests to do “problem solving” sessions with other parts of the company.  Typically, you run these projects through the same analytical process you used with the web site: Define objectives, create / get buy off on KPIs, measure baselines, develop ideas for testing, test and measure the results of those tests.

Recently, you were working on a project for customer service, trying to develop / improve KPI’s for the measurement of performance in the call center, which has a very rich data set.  This data is surprisingly similar in many ways to the traffic data set from the web site.  A phone call is very much like a visit; it has a duration, it typically has a number of steps like a web site funnel, and the steps end with accomplishing or not accomplishing a goal.  The call center is trying to evolve from relying on simple metrics that score only “efficiency” to a KPI that better balances efficiency and a good customer experience.

Your web site work has lately focused on a new campaign that the marketing folks are very proud of.  It’s blowing the doors off anything they have ever done in terms of response, thanks in large part to your analytical work on promotions.  The success of this program is widely known throughout the company, as is your role in the success.

On a Friday afternoon, you find yourself with some free time to devote to the customer service KPI project.  While examining the call center data set, a remarkable possibility presents itself.

Using a new, experimental KPI you have developed that balances efficiency and customer experience in the call center, it appears that every time marketing drops this new, highly successful campaign, there is a dramatic negative spike in this new call center KPI.  The correlation between the marketing campaign drop and negative spike in the call center KPI is extremely high, leaving no doubt in your mind that that there is a causal relationship between the two events.  There is always some negative impact on the call center when a campaign drops, but nothing like the magnitude of the impact caused by this most successful campaign.  Nothing about the campaign execution – for example, the volume of the drop – would lead one to conclude it should cause problems in the call center.

What would your next steps be?  What should you do with this knowledge that (as far as you know) only you possess?  The topics below might be worth touching on:

Topic 1.  You work for IT, and your main internal client is marketing.  The customer service analysis is a side project.  What responsibility do you have for resolving the apparent conflict between optimizing marketing and optimizing the call center?  Is it your responsibility to try and “optimize the company” across all the business units by providing this kind of information?

Topic 2.  One alternative would be to try and alter the new KPI (which you feel is very, very good) so that it masks the effect of the marketing campaign on the call center.  This would reduce potential internal conflict, for sure, but would result in a weaker, less trustworthy KPI for the call center.  Would you consider this route? What if your main client (Marketing) suggests you “tweak the customer service KPI a little bit to help us out on this”, what would you say?

Topic 3.  What kind of action plan can you imagine for trying to resolve the apparent conflict between the success of the campaign and the performance of the call center?  Would you call a meeting first or speak privately to some folks and discuss a potential meeting?  Who would you speak to 1st, 2nd, and 3rd given your ties to IT, marketing, and customer service?  Who would be invited to the first meeting?

Looking forward to hearing your ideas on this or similar situations you have faced.  If you’re relating a real analytical culture war story, you might think about changing the names to protect the innocent!

And / Or, if you’d like to share your story interactively with the Course 4 students in their Cafe’ (chat), we’d love to have you as a guest.  FYI, the students are primarily adults who are already working with web analytics as part of their job who now are faced with a need to upgrade their skills.  Let me know if you are interested in sharing your story with them – you can use the “Email Jim” link below to contact me.