The Analyst’s Enigma

Some of you might know that I produced the†Web Analytics Association’s†continuing education courses on Web Analytics (with Co-Chair Raquel Collins and a ton of great contributors).† One†part of this†project I am†very proud of is the 4th Course, Creating and Managing the Analytical Business Culture.† This course was a real “stretch” goal because initially, the Education Committee†thought there simply would not be enough material to create an entire course on this subject.† But we did it, and more importantly, the students really love it, because it addresses the “politics of analytics” they face every day.

One of the†exercises in this 4th course is called The Analyst’s Enigma.† It always generates a lot of intense discussion and without giving away too much of the actual exercise, I thought you might like†to either comment o†how you would handle this situation or¬†share your own similar war stories and how you handled them.

By the way, it’s based on an actual†situation you may well have experience with (or will in the future) as web analytics folks spread their wings and†bring their business optimization skills to the rest of the company.†Here goes:

The Analyst’s Enigma

You are the manager of web analytics at a large public company.† The “web stuff” in the company started out in IT, so due to legacy reasons you are part of IT, but your internal client is really the marketing department, who is now in charge of the web site.† About 95% of your time and effort is spent working with marketing to optimize the web site, a project that has been very successful.

Over time, your success with the web site optimization process and as an analyst has been recognized in the company, and you occasionally get requests to do “problem solving” sessions with other parts of the company.† Typically, you run these projects through the same analytical process you used with the web site: Define objectives, create / get buy off on KPIs, measure baselines, develop ideas for testing, test and measure the results of those tests.

Recently, you were working on a project for customer service, trying to develop / improve KPI’s for the measurement of performance in the call center, which has a very rich data set.† This data is surprisingly similar in many ways to the traffic data set from the web site.† A phone call is very much like a visit; it has a duration, it typically has a number of steps like a web site funnel, and the steps end with accomplishing or not accomplishing a goal.† The call center is trying to evolve from relying on simple metrics that score only “efficiency” to a KPI that better balances efficiency and a good customer experience.

Your web site work has lately focused on a new campaign that the marketing folks are very proud of.† It’s blowing the doors off anything they have ever done in terms of response, thanks in large part to your analytical work on promotions.† The success of this program is widely known throughout the company, as is your role in the success.

On a Friday afternoon, you find yourself with some free time to devote to the customer service KPI project.† While examining the call center data set, a remarkable possibility presents itself.

Using a new, experimental KPI you have developed that balances efficiency and customer experience in the call center, it appears that every time marketing drops this new, highly successful campaign, there is a dramatic negative spike in this new call center KPI.† The correlation between the marketing campaign drop and negative spike in the call center KPI is extremely high, leaving no doubt in your mind that that there is a causal relationship between the two events.† There is always some negative impact on the call center when a campaign drops, but nothing like the magnitude of the impact caused by this most successful campaign.† Nothing about the campaign execution – for example, the volume of the drop – would lead one to conclude it should cause problems in the call center.

What would your next steps†be?† What should you do with this knowledge that (as far as you know) only you possess?† The topics below might be worth touching on:

Topic 1.† You work for IT, and your main internal client is marketing.† The customer service analysis is a side project.† What responsibility do you have for resolving the apparent conflict between optimizing marketing and optimizing the call center?† Is it your responsibility to try and “optimize the company” across all the business units by providing this kind of information?

Topic 2.† One alternative would be to try and alter the new KPI (which you feel is very, very good) so that it masks the effect of the marketing campaign on the call center.† This would reduce potential internal conflict, for sure, but would result in a weaker, less trustworthy KPI for the call center.† Would you consider this route?†What if your main client (Marketing) suggests you “tweak†the customer service KPI a little bit to help us out on this”, what would you say?

Topic 3.† What kind of action plan can you imagine for trying to resolve the apparent conflict between the success of the campaign and the performance of the call center?† Would you call a meeting first or speak privately to some folks and discuss a potential meeting?† Who would you speak to 1st, 2nd, and 3rd given your ties to IT, marketing, and customer service?† Who would be invited to the first meeting?

Looking forward to hearing your ideas on this or similar situations you have faced.† If you’re relating a real†analytical culture†war story, you might¬†think about changing the names to protect the innocent!

And / Or, if you’d like to†share your story interactively with the Course 4 students in their†Cafe’ (chat), we’d love to have you as a guest.† FYI, the students†are†primarily adults†who†are already working with web analytics as part of their job†who now are faced with a need to upgrade their skills.† Let me know if you are interested in¬†sharing your story with them†- you can use the†”Email Jim” link below to contact me.

3 thoughts on “The Analyst’s Enigma

  1. Jim —

    TD Waterhouse (a few years ago, prior to the Ameritrade merger) discovered exactly what you described. They discovered it a little differently — they wanted to a new online search engine. To optimize the system, they went into the call center to understand what the most frequently asked questions were, and determined which ones they could adequately address online. They built the online capability, and then publicized it — both through online messages and on statements (not sure if they did it through call holding messages).

    They were successful at migrating calls out of the call center over to the online channel. But there was no animosity on the part of the call center and there were no mass layoffs in the call center.

    Because the call center was involved in this from the start, they had to ability to plan and contribute to the shift in staffing, and was able to take a number of reps and focus them on higher value (ie, sales-related) calls than lower value account status calls.

    Your theoretical analyst should have nothing to fear in bringing his or her analysis to light if s/he appropriately involves the right people.

  2. Your theoretical analyst should have nothing to fear in bringing his or her analysis to light if s/he appropriately involves the right people.

    Yea, and that’s the key to the exercise – appropriately.¬†

    1.  Who do you talk to

    2. In what order, and

    3.  About what subjects

    really can have an effect on how these things play out.¬† Many analytical people would¬†rather just stick with the number crunching and have “managers” deal with the 7 Sins of Performance Management.¬† But increasingly, at least on the web analytics side, folks are developing a lot of business skills.¬†¬†The “artful presentation of bad news” is one of those required skills if you are going to be an active rather than passive analytical player – which is what we encourage.

    My wife is one of the “old school” analysts – she loves to crunch the numbers, but hates the “confrontation” associated with uncovery.¬† I am very much the opposite, perhaps too aggressive in seeking who is behind the Root Cause.¬† Best stance is in the middle somewhere, and navigating those waters is what we try to teach in the Course.

    Great example, thanks for sharing Ron!

  3. And if I knew “who to talk to…in what order… and about what subjects”… perhaps I would’ve made it a little further up the org chart in my career!

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.