Lab Store: Automating Worst Practices

The news that Omniture has acquired Touch Clarity is shaking up the world of web analytics a bit.  Machine automation has always been a very sexy sell for software companies.  The problem is people think it’s a magic bullet and often end up using these tools to their disadvantage because they do not have the experience to really understand how to use the tools properly.  Then they get caught in trap of Reporting versus Analysis.

Here is a real world example from the Lab Store.  I am constantly fighting the Google AdWords A/B split testing algorithm for rotating ads.  Google almost always picks the wrong ad to run more frequently so I have to force it to run 50 / 50 in order to get accurate results.  How do I know Google is picking the wrong ad?  Because I have seen thousands of such tests, online and off, and I have a “feel” for these things based on my background in Database Marketing, Consumer Behavior and Psychology.  In each case where Google has picked one ad over another, and where I have forced it to then run the ads 50 / 50, it ends up I was right – Google picked the ad that generated the least profit per dollar of PPC spend as “best” and demoted the more profitable ad until it was not running at all.

Why does this happen?  Because Google isn’t smart enough to understand the complexity of the customer behavior in the Lab Store – and it can’t be, given the number of clients it has.  If you have done a lot of this kind of testing, you know that often the campaign with the highest response rate generates the lowest quality customers.  While these campaigns were running, I could see that the visitors generated by the campaigns Google picked as “best” were actually inferior to the visitors generated by the campaigns Google demoted, using a variety of metrics other than conversion (primarily Recency).  In other words, I was able to predict Google was doing the wrong thing by looking at the Customer LifeCycle. When I forced Google to run the ads 50 / 50 to give the demoted ads a chance, I was proven right – the campaigns Google demoted had a 90-day ROMI averaging 2.1 times higher than the campaigns Google promoted.

Look, I know these are software companies and their sole purpose in life is to create the next big thing and sell their software into it.  That’s fine, and frankly, I hope they are successful in doing it, because it will create a tremendous amount of business down the road for database marketing consultants as “machine optimization” hits the wall and companies need to be rescued from the results of it.  Just like they had to be rescued from demographic clustering in the 80’s and data mining in the 90’s.

People are always looking for the easy way out, and it ends up costing them more in the long run because they don’t really understand what the tool does and does not do. Perhaps that is simply the state of Marketing today.  So be it…

If you are an analyst and you see a black-box test result that simply does not make any sense based on your past experience, I encourage you to question the result, find a way to test it outside the system.  Learn why, because this kind of incident usually will lead to a shattering of some myth or bias you will be most happy to fully understand!

3 thoughts on “Lab Store: Automating Worst Practices

  1. Hi Jim,
    It is nice to talk with you again; we know each other from my 6 years running marketing at WebTrends. I hope you are well. I read your recent post and thought it appropriate to address some of the points you make, as a fellow analytically-driven marketer with a unique perspective. I discovered Touch Clarity last Spring after I left WebTrends and initially, I too had great suspicion about the ability to automate the serving of the most valuable message to each visitor. I think all of us who have been deep in the analytics world hear this story and say “no way can that truly work.” But then I spent many hours with Paul Phillips the founder of Touch Clarity, who is a multiple award-winner of the most prestigious direct marketing data-mining competitions in the world, the PhD’s on the modeling team, and the client services team who walked me through an actual customer implementation, and showed me their uplift reports. Was I convinced? Yes, and in fact, I then bet my career that what they were doing was the next evolution of Interactive Marketing. And now 10 months later, it can honestly say it was the right bet. It really does work, and I personally offer my time to walk you through how the system has been designed to accommodate everything you mention, such as the fact that the content delivery can be optimized to achieve not only the highest click-throughs, but even better to achieve increased conversion, or to maximize revenue or profit.
    The modeling algorithms that drive all of this in an automated manner consider hundreds of variables in their scoring of visitors and in deciding which content to serve. All past behavior and current behavior of each visitor is considered…including recency and frequency of past visits, previous content exposures, referral sources and keywords, temporal and technographic and geo-location factors along with all of the other web analytics variables collected anonymously. Each one of these variables has a small amount of predictive power which collectively add up to a clear determination of the best content to serve to that visitor at that moment. In addition, we do consider sales cycles to watch for conversions or purchases that occur over several visit cycles after the first offer or content engagement. And you are absolutely right—other so-called “automated” systems that consider a few variables are apt to deliver the wrong answers, and may optimize simple click-throughs but over the long term degrade business results. That is why business rules engines that hard-wire a few simple decisions will fail.
    So how else does Touch Clarity continuously ensure the best decisioning of content serving? The system always targets versus a concurrent control group, which can be a champion/challenge scenario or a random serve. The system will automatically detect when perhaps there isn’t a clear best piece of content to serve and then test more until it knows for certain the best content to serve. Response curves vary several times every single day, so the system needs to be able to detect those curve shapes and change how it serves content accordingly.
    The solution also gives the marketing team full view of all of this through real-time reporting of the lift generated versus the control. So they know exactly the ROI of the targeting solution, at all times.
    So yes, this sounds amazing. It’s why the folks at Omniture made an investment to acquire the company rather than just partner. Knowing the analytics space very well, I believe it was a brilliant move and a natural extension for web analytics in general.
    Jim, we absolutely see marketing consultants and interactive agencies as being key partners with this technology. We automate the optimization and targeting of the best content from a catalog of candidate content options, but our customers must determine the content strategy–which products and product categories to promote, the best manner in which to promote them, which design styles to build, which offers to test, which calls to action, etc. etc. There is even more of an opportunity to work in both the left and right brain arenas with the Touch Clarity technology, and the best thing, is that it’s no longer an arbitrary decision or a political decision about what to put on the home page or key product pages. Marketers can get back to marketing strategy and creating different versions of creative and letting advanced mathematics and computers process untold amounts of data in milliseconds, considering hundreds of factors, to choose the best piece of content for that visitor at that time, and then be absolutely rigorous in making sure it is right, based on hitting a specific objective that the marketer defined. Perhaps accountants feared that Lotus 123 and Excel which automated the calculations of thousands of rows and columns of numbers in seconds would put them out of work, but frankly, it allowed finance professional to become more strategic business advisors and today we see the CFO as a key player of business teams, because they were freed up along time ago to focus on more strategic decisions than adding up numbers.
    I look forward to continued dialogue about this subject. Let me know when we can arrange a personal briefing.
    Best,
    Brent Hieggelke
    Chief Marketing Officer
    Touch Clarity from Omniture

  2. Jim, don’t be silly… :) AdWords optimizes for click-through, plain and simple. That’s great if your a GOOG shareholder, probably not so great if you’re an advertiser.

    But take note of the silver-lining: you can drill down within Google Analytics “Keyword Specific Testing” report to reveal which creatives are driving profit. But alas the built-in AdWords optimizer doesn’t seem to use this data.

    Eric

  3. Eric – you know what’s funny? If you run two identical AdWords campaigns with identical landing pages Google will choose one as the “winner”? I can’t figure that one out…

    Brent,

    We have known each other for quite some time and have both played (along with others) a major role in the establishment of web analytics as a credible and proven way to increase the productivity of a web site. The providers of software and services in this space have managed (for the most part) to keep the dial turned down on puffery and let the facts speak for themselves. Please understand that the comments I have made (and I think this is true for Future Now and Shane as well) don’t challenge the Touch Clarity technology. Math is math; I think everyone would agree “math works”.

    I think what people are concerned about is statements like “delivers automated revenue and profit lift”. You push a button and make money. It’s pretty clear that no software, no matter how smart, delivers automated revenue and profit lift. As you said in your comment above:

    “Our customers must determine the content strategy–which products and product categories to promote, the best manner in which to promote them, which design styles to build, which offers to test, which calls to action, etc. etc.”

    None of that is “automated”, right? It takes people who know something about the business to come up with a good content strategy. And that’s all I am saying. I’m hoping we can continue the great reputation web analytics has for delivering value as we enter the next generation of tools like Touch Clarity, and don’t get ourselves caught in the hype trap so many other technology areas have.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.