The Measuring Engagement series starts here.  For a clickable index of the 5 part Measuring Engagement series, look here. 

Last time we addressed the topic of measuring Engagement – and attributing actual Value to it – we were looking at visitors generated by various campaigns.  Here is what the Frequency (average number of visits) and Recency (average days since last visit) look like in a web analytics interface:

And here is what the Campaigns, numbered 1 – 16, look like in the Current Value / Potential Value Map:

Quadrant 1 contains campaigns generating visitors with both high Current Value and high Potential Value – these are the campaigns deserving more investment because the visitors created generate highest value to the company now, and have the highest likelihood to generate more value in the future (are the most Engaged).  If you’d like to know more about what metrics drive the Map and how it was created, see here.

Beyond Campaigns, how else can we use the Current Value / Potential Value Map?

Search Phrases

One of the more interesting uses is looking at search phrases as the “campaigns”.  Search marketers, especially PPC folks, are often victims of initial conversion rate-itis, where campaigns are managed and funded based on a short-term conversion rate.  To be fair, often this is a systems integration problem more than anything else – there simply is not enough “visibility” in the out weeks to determine if longer-term conversion to final goal is occurring.  This is common where there is not a clean integration between web analytics and the back-end commerce system, for example.

Using the Customer Value Map with search phrases provides you with a way to imply a future conversion and balance out some of the decision making on short-term conversion.  If you know a certain search phrase is generating visitors who visit Frequently and are still Recent in their visit behavior (Quadrant 1), you can imply this phrase is going to be more profitable than a phrase generating visitors who end up in Quadrant 4.  For an example of this idea in action, see here. 

Likewise, let’s say you’ve optimized the heck out of all PPC campaigns as far as copy, landing page navigation, etc. and still have a number of phrases that are “breaking even” on an ROI basis.  But some of these break-even campaigns consistently deliver visitors who end up in Quadrant 1.  The last campaigns I would kill are the ones delivering visitors who end up in Quadrant 1, since these visitors have the highest Potential Value.  Kill Quadrant 4’s first, then 3’s, then 2’s to see if you can get where you need to go in the overall ROMI mix.  Then do anything you can (including fishing through databases / logs manually, if need be) to find out if those Quadrant 1’s are really not paying out – I’d bet something is missing, there is a break in the logic / code somewhere that is not giving credit where credit is due.

Navigation / Functionality

Before we get into this area, let’s step back a minute for a global thought. 

This Retention / Engagement analysis stuff may seem oddly strange to you, and if it does, this is probably the reason: what is most important to measure in this area is what does not happen. 

Think about it.  This is not what you are used to in web analytics (or most other transactional analysis) – you are always focusing on what did happen.  How many visitors, clicks, conversions, etc. happened?  But I ask you this: in terms of Objective / Action, where would you want to take action in the Engagement area, where would the highest payout be?  Right.  Not with the Visitors who are already Engaged, but with those who are becoming less Engaged – where something is not happening.

Keep that in mind as we go through the next section…

Has this ever happened to you?  Your revenue KPI’s start sinking, gradually at first, and then at an increasing rate.  You run around trying to figure out what the problem is – campaigns, changes in natural ranking, competitor activity, whatever.  You’re pulling your hair out because it doesn’t make any sense – everything is tracking “normal”, right?  No changes in the past few days, or even weeks?  Right.  So, what the heck is going on?

Understanding the Volume of traffic by segment to your site is a given.  But what happens to visitor Value segments after their first visit cycle is important as well.  I can’t tell you how many times I have seen people screw themselves over the longer run because they are tracking / optimizing for Current Value rather than both Current and Potential Value.  This is a particularly important idea when you are testing new navigation / functionality and content or products, because it’s not only Campaigns that determine the long-term quality of visitors, but also the site itself.

Here’s an example.  Let’s say you have a simple visitor value segmentation of visitors during the past 12 months that divides the Current Value of Visitors into 2 groups – Frequency over 50 Visits and under 50 Visits.  Further, you divide Potential Value (Engagement) into 2 groups – Recency of Visit within 2 months and over 2 Months ago.  You end up with a 2 x 2 Visitor Value Map that looks something like this, with percentage of the 12 month visitor base listed in each Quadrant:

(Analysts: This simple data set, the first time you present it, may cause some rapid heart beats,  Trust me, most every site looks about like this – the majority of Visitors are in Quadrant 4 – have only visited a few times and have not been back lately.  What’s a few rapid heartbeats among friends anyway??  Gulp…  Hey, you’re an analyst, you’re used to this kind of thing!)

In the chart above, we see 10% of your Visitors are in Q1 (Quadrant 1) – at least 50 visits, Last Visit within 2 Months.  These are the 10% of your Visitors who probably drive the majority of your revenue, the “rocket fuel” visitors.  Q3 is where former best Visitors end up – they have high Frequency / Current Value but have abandoned visiting the site.  If you’re not clear how time since Last Visit date correlates to site abandonment, see here.

Now, let’s say you make a major change in navigation on the site.  Traffic flow to the site remains the same; all the same campaigns are running and everything seems normal.  Hopefully, conversion even goes up (that’s why you redesigned the nav, right?) 

A couple of months later, all of a sudden your revenue per visitor or visit metrics start to slip. 

Thankfully, you have been keeping track of the Percentage of Visitors in each Quadrant of your Customer Value Map over time (phew!) – I wonder what that looks like?  Here is what you find:

The Quadrant 1 Visitor segment (Top Graph, dark line) is shrinking; it has dropped from 10% of the visitor base to 6% or so over a 7 month period.  Doesn’t sound like much, right?  That is, until you remember that these Quad 1 rocket fuel visitors are responsible for a very significant portion of your revenue.  This means, of course, that your revenue per visitor follows the shrinking Quad 1 population right down the curve, as shown in the Bottom graph above.

Think about it.  In terms of gross numbers on the site, you would hardly notice a change like this in any of the “did happen” metrics.  Traffic and conversion, traffic and conversion, all just chugging along, right?  But this change in a small yet powerful group of Visitors significantly affects your Revenue KPI’s – because something did not happen.

Where are these Quad 1 visitors going?  Well, they are becoming dormant – they are moving into Quad 3 – high Frequency but poor Recency (Engagement).  It’s really the only place they can go; most can’t move to Q2 or Q4 because they have high Current Value as they start to move.  So as the population of Q1 shrinks, the population of Q3 rises, as seen in the Top chart.

What you are seeing in the chart above is a tangible visual representation of Best Visitor defection – visits not happening among most Valuable Visitors – that is hard to dispute.  Can you say Engagement Dashboard?

Then why is this happening?  I’d bet on the navigation change.  The problem is, of course, that unless you have a chart like the one above, it will be difficult to prove this idea to anybody, since the drop in the revenue KPI’s lagged the navigation change by such a long time, and all else remains consistent.

The fact is, you changed your “product” – the web site.  For some reason, the site simply does not generate or retain high value Quad 1 visitors like it used to.  Perhaps you pissed off the current Quad 1 Visitors with your changes.  Maybe the parts of the site that create new Quad 1 visitors are now buried in the new navigation, so up-and-coming Best Visitors (Quadrant 2) never find these high value creation areas. 

Did you bury sections of the site considered “low volume” in the navigation?  Better check that idea, because the low volume areas (uniquely targeted areas?) often create the highest value visitors.  You can check on this by running a Current Value / Potential Value Visitor Map for each Content Group – hopefully, before you make any changes to the web site!

Next time we visit this topic, we will look at Customers – those good folks who actually pay money to support a web operation.  If your web analytics tool does not support Visitor Frequency and Recency, you can still use the same Current Value / Potential Value model to manage Engagement through your customer database.

As always, your comments and questions appreciated…

The next post in this series on Measuring Engagement is here.

The Measuring Engagement series starts here.  For a clickable index of the 5 part Measuring Engagement series, look here. 

OK, now that you have (way too much?) background, let’s get to the “how to” on this Recency / Engagement stuff.

Recall that you can plot value-creating visitor / customer actions on a common platform and create a “customer engagement map”.  Decide what visitor variable or action representing value to the firm you want to map – this is Current Value.  Number of Blog posts or Comments, Visits, purchases, page views, visits to a certain content area, sign-ups for a newsletter, opens, clicks, you name it – any action that either creates value directly or represents value to to the firm.

Then take a look at how Recent the visitors or segments are in terms of accomplishing the action; this is Potential Value (if you’re not following this reasoning, see here).  Plot Current Value and Potential Value of each segment and you have a very easy to understand “map” of where you are – and where you could go – in terms of increasing customer value / engagement.  Each quadrant of the map has a general marketing objective in terms of allocating resources – Keep the Customer, Grow the Customer, Question Spend on the Customer.

Why should you do this kind of mapping?  Two reasons – consistency and repeatability.

Consistency, meaning the world of web analytics is messy enough and you want to introduce some rigor to the thought process and decision making when looking at various aspects of optimizing visitor / customer value.  Having a simple model to compare the various aspects of engagement ensures a level playing field for all – including the comparison of offline and online efforts.  This is not a web only model; it works in the “real world” as well.

Repeatability, meaning Recency is an incredibly stable behavioral metric that delivers time and time again, in the exact same way, over and over.  You can bet your marketing or operational dollar with confidence every time on Recency.  The more Recently a segment has accomplished any of your goals from above, the more likely they are to accomplish that goal again – either by themselves or due to stimulation on your part.

Let’s take a look at what this looks like with Campaign data.  Here we have a series of new campaigns that all started at the same time (Campaign names have been blanked out to protect the client) with Frequency of Visit data as the Current Value and Average Recency of Visit in each Campaign as Potential Value.  In this case, we are looking at the First or Initial Campaign the visitor was exposed to:

 

For this web site, Visits are the action that generates value for the firm.  These 16 Campaigns are sorted by Frequency of Visit – the Current Value of visitors generated by each Campaign.  The second value is the Average Recency of visitors from that Campaign – the Potential Value of the Campaign.  You can see that Campaign 4 generates visitors with High Frequency and very Low Recency – this Campaign is generating Visitors that not only add value to the firm, but are highly likely to add value in the future.  In other words, they are both high value and engaged.

What would these campaigns look like on the Current Value / Potential Value map?  Glad you asked; here they are ranked on a relative basis to each other:

Since the ID numbers 1 – 16 for the campaigns were sorted by Frequency / Current Value to begin with, campaign ranking on Current Value starts at the top of the chart and falls to the bottom.  Potential Value grows as you move from left to the right.  Each Quadrant has a red underlined number ranking the relative desirability of investing more capital in the campaigns from that Quadrant.

In Quadrant 1, we have the “rocket fuel” campaigns.  These campaigns are generating visitors who both keep coming back Frequently and are Engaged - visitors with high value to the firm now and in the future.  These visitors are likely in the “20% of visitors generating 80% of visits” component of the visitor base.  In Quadrant 4, you have campaigns generating visitors who don’t have much value now and won’t have much value in the future.  In general, you want to reallocate spend on Campaigns in Quadrant 4 towards spend on Campaigns in Quadrant 1.  This will optimize your Campaign throughput and longer term ROMI; you want to “gun” Campaigns 2, 4 and 6.

In Quadrant 2 we have campaigns that don’t generate a lot of Current Value but have high Potential Value; these visitors are “potential up and comers”.  Please note: due to the lower Frequency / Current Value, people often kill these Campaigns before they have a chance to blossom.  This is usually a very bad idea and often can gut the profitability of the overall effort.  The Recency / Potential Value component tells you these visitors are engaged; they simply have not delivered much Current Value so far; if at all possible, let these campaigns run.

In Quadrant 3 we have the “early bloomers”; high Current Value and low Potential Value.  Ironically, these are the campaigns people tend to invest more money in due to the early activity, often robbing from Campaigns in Quadrant 2.  This can be misguided, depending on what your end objective is.  If you want to build an engaged visitor base, this is not the place to invest.  If you just want to generate short-term “activity”, then these are the campaigns you want.

But budget allocation isn’t the only thing going on here.  Knowing how these campaigns are mapping across the visitor value grid, you want to ask yourself these additional questions:

1.  What is similar about campaigns in Quadrants 1 & 2 that is different from Campaigns in Quadrants 3 & 4?  In other words, what is it that generates engaged visitors with high Potential Value?  Is it campaign media, copy, offer, channel, product, content area of the site they are sent to?  What are the drivers of this behavior?

2.  What is similar about campaigns in Quadrants 1 & 3 that is different from Campaigns in Quadrants 2 & 4?  In other words, what is it that generates high Frequency visitors with high Current Value?  Is it campaign media, copy, offer, channel, product, content area of the site they are sent to?  What are the drivers of this behavior?

This is where the ideas of Consistency and Repeatability driven by the Current Value / Potential Value platform come into play.  If you can distill why certain campaigns generate visitors that end up in each Quandrant, you can:

1.  Rely on the same campaigns to generate visitors that will always land in a certain Quadrant – you get results that are Consistent instead of wondering why things happen the way they do, which often is the case when using segmentation based on demographics, product affinity, etc.  Behavior predicts Behavior, as long as you are tracking actual behavior.

2. Know that when you take these drivers of behavior out of the current campaigns and create new campaigns, your results should be as expected.  You will be able to Repeat your success in other types of campaigns as well as in related ideas like web site copy, landing page copy, merchandising, and so forth.

You can also use these drivers to tweak and improve the behavioral performance of campaigns like 1, 3, 8, and 9, eventually moving them from borderline to solidly into Quadrant 1.

Are you with me?

For those of you looking to integrate online data with offline, let me suggest that a simple “Quadrant Tag” of 1 – 4 for a customer would contain a ton of actionable data about the web behavior of that customer in a very small space.  You could create a “Master Tag” for the most important web KPI or export a series of Quadrant Tags for a variety of KPI’s.  There really is no need to send a lot of detail to CRM if you can send only the most actionable keys, which can be used for trigger-based responses in either campaign automation or call center scripts.  There is no better high-level summary of a customer’s web activity than a single digit that represents the Current and Potential Value of the customer on the web.

By the way, this model generally tracks with the valuation model folks on Wall Street use to value direct and database marketing companies.  Wall Street wants to know two things in this area: how much have customers spent, and how Recently did they spend it? 

This is why you so often hear metrics like “12 month Active Customers” used to describe Amazon or eBay.  Investors want to know not only the Current Value of the customer base, but also the Potential Value.  The 12-month model is a bit “slow” for my tastes, especially when you are talking about the web.  It’s much better to use the raw Recency stats to build out the Value Model above and find out exactly what the heck is going on. 

You know anybody that makes online campaign decisions based on a 12-month test window?

Recency is in fact a very simple, single variable, predictive model.  I’ll bet you any money if you go out and create a complex, multi-variate regression model for predicting “likelihood to repeat action” that Recency will be in there right at the top.  Not to say those more complex models are not valuable, you know; they are – as long as you have the capability to execute on them.  But Recency by itself is very powerful and you can use it right now, both online and offline, to measure engagement and optimize the Potential Value of your customer base.

As always. your comments and questions on the above are appreciated!  In the near future, we’ll look at more applications beyond Campaigns for using the very same mapping of Current and Potential Value for Visitors and Customers.

The next post in this series on Measuring Engagement is here.