Category Archives: Driller Q & A

PRIZM Clusters Not as Predictive as Behavior

DataBase Marketing, Driller Q & A, Newsletters

Jim answers questions from fellow Drillers
(More questions with answers here, Work Overview here, Index of concepts here)

PRIZM Clusters Not as Predictive as Behavior

Q: I am on an interesting project (and my first DB Mktg one): the client has a large loyalty program, and loves his PRIZM clusters. However, when I told him a little more about Recency and suggest that we spread all members across based on it, he was surprised to see that his PRIZM segments were not a predictive indicator at all!

A: Yes, and here is something many people don’t realize about PRIZM and other geo-demo programs, including census-driven. They were developed for site location – where should I put my Burger King, where should I put my mall? They are incredibly useful for this. However, think about all the sample size discussions for web analytics related to A/B testing, and now imagine what your PRIZM cluster looks like.

In most cases, you are talking about 1 or maybe 2 records in a geo location – what is the likelihood these households reflect the overall “label” of the PRIZM cluster? Combine this with the fact that for customer analysis, demographics are generally descriptive or suggestive but not nearly as predictive as behavior and you have a bit of a mess.

Here’s a test for you. It only requires rough knowledge of your neighbors, so should not be very difficult (for most people!)

1. What is your “demographic”?
2. If you were to walk around the block and knock on doors, how many households would you find that are “in your demographic”?

Right. Maybe a handful, unless you live in a brand new housing development or other special situation.  Now think about walking your zip code, or walking out 10 blocks or so from your house in any direction, and knocking on doors. Do you find most of these people are in the same demographic as you are? Did you ever find the “cluster average” neighbor?

We certainly know from web analytics that dealing with “averages” can be very dangerous indeed. So too with taking a demographic “average” of a zip or other area and tying it to a specific household. The model falls apart at the household level of granularity.

So now what to you think of all those websites and services that claim to know demographics based on a zip code they captured?

Now, if you think about an e-commerce database, with most records being one of a very few in a zip or cluster, you can see how the cluster demos would really break down at the household level.

Again, nothing wrong with using these geo-demo programs for what they were intended to be used for. When you are looking for a mall location or doing urban planning they can be very helpful. But the match rates at the individual household level are poor.

Couple this with the fact that e-commerce folks are usually looking for behavior from customers, and the fact demographics are not generally predictive of behavior by themselves, and you have yourself analytical stew.

Better than nothing?  Absolutely, and for customer acquisition, sometimes all you can get. Best you can be? Not if you have the behavioral records of customers. In fact, what we often see is a skew in the demographics being called “predictive” when the underlying behaviorals are driving action.

In other words, let’s say a series of campaigns generates buyers with a particular demo skew. A high percentage of these Recent responders then respond to the next promotion. If you look just at the demos, you would see a trend and declare the demos are “predictive” of response, even though they are incidental to the underlying Recency behavior.

I suspect something like this was going on with your client. Not looking at behavior, over time the client becomes convinced that the PRIZM clusters are predictive, when for some reason they are simply coincident in a way with the greater power of the behavioral metrics. Given the client has behavioral data, that should be the first line of segmentation.

Q: After reading you for some years, I now understand how one must be very careful with psycho-demographics.

A: Well, at least one person is listening!  And now you have seen how this works right before your very own eyes.

I think this situation is really a function of Marketers in general being “brought up” in the world of branding / customer acquisition. Most Marketers come up through the ranks “buying media” or some other marketing activity that focuses on demographics to describe the customer. And most of the college courses and reading material available focus on this function, so even the IT-oriented folks in online marketing end up learning that demographics are really important. And they can be, when you don’t know anything about your target.

Then the world flips upside down on you, and now people are looking at customer marketing, and that’s a whole different ballgame. The desired outcome is “action” that can be measured and the “individual” is the source of that outcome, as opposed to “impressions” and “audience”.

In the past, if your tried and true weapon of choice for targeting was demographics, that is what you reach for as you enter into the customer marketing battle.Problem is, it’s just not the best weapon for that particular marketing engagement.

Jim

Get the book at Booklocker.com

Find Out Specifically What is in the Book

Learn Customer Marketing Concepts and Metrics (site article list)

Download the first 9 chapters of the Drilling Down book: PDF 

Tracking UnTrackable Campaigns

Analytical Culture, DataBase Marketing, Driller Q & A, Newsletters

Jim answers questions from fellow Drillers
(More questions with answers here, Work Overview here, Index of concepts here)

(Any Tracking is Better than No Tracking at All)

Q: I have a background in direct marketing and the measurement of campaigns using defined sources.  Now I am working at a technical or 2 year college and I’m trying to use my previous knowledge and experience to measure our return on marketing efforts in recruiting new students and converting them from prospects to enrolled students at the school.

I’m looking to measure events and all of the advertising and promotion used to communicate those events as well as other campaigns. It’s difficult to measure since it may include newspaper ads, direct mail, posters, etc. – not just direct mail with a measurable list to refer to for tracking response.

Plus, trying to get the admissions folks to track leads can be difficult. They also want to track return on publications (brochures, flyers, etc.)….not sure how one would do that if they are not mailed to a given measurable list?

I’m looking to track ROI based on the whole equation….from the # of inquiries who came to an event or responded to whatever all the way to whether they matriculated and enrolled in courses.

A: I can feel your pain!

I don’t think there are any easy answers to this. You could simply measure what you can measure through the traditional direct methods you are familiar with, and let the rest “ride”. Or, you can try to selectively determine, as best you can, what the value of all these other activities is by engaging in some kind of testing. This will take some institutional willpower and is probably something you can’t do on your own. In other words, I encourage you to start evangelizing the next generation of marketing measurement at the school.

From your title, I perceive you don’t “report to” marketing, but some higher institutional level responsible for Quality and perhaps Productivity / Accountability for funds that are spent? It sounds to me like your unit might perhaps report into the Financial area of the school at a higher level, and if so, that’s good!

I think I would simply start the conversation with folks in the financial area about some of these issues, and see if you can create some simple tests to get some “direction” on the contribution of the various marketing outlets.

For example, every publication should contain some kind of tracking device. Sometimes you have to be creative with this idea and it won’t always be accurate, but it’s better than nothing at all. If response is generally by phone, then try to get a unique phone number for newspaper ads, brochures, etc. If response is driven to the web site, get unique URL’s put on each document. If response is filling out a sheet or card at an event, have them numbered or coded in some way. Then of course, you need to get the response information – number of phone calls to each number, number of visits to each unique web site URL, number of response cards mailed or turned in.

Implementing a program like this, then finding and getting access to the response info may not be easy, and that’s why it would help to have a “higher power”, particularly a financial one, backing this effort. It’s pretty amazing what people will do when, for example, the people who control the budget for an area say, “You will participate in this tracking program”.

Alternatively, you could go with a test / control kind of scenario where during a quarter, you leave out one particular marketing effort and see if there is an impact on overall Marketing Productivity. This is more of a “marketing mix” kind of approach and not without some problems, including proving the missing marketing effort was responsible or not for changes in Productivity. You have to think about how you might pin these issues down in advance – for example, do you have good baselines for “normal” activity?

Whether or not you decide to pursue all of this is somewhat of a personal choice. Some analysts simply don’t think it is their “job” to help create measurable structures – they only measure what can be measured. Others see the difficulty as a challenge, and want to help build out the structure. Clearly, if you are going to eventually be responsible for measurement, being a part of the team constructing the measurement paths is a real advantage to you. It will involve some politics, but analytics always involves politics at some level. I encourage you to seek out the support you need to make this work.

If there is “pressure” for measurement, someone wants it to happen. Start by finding these people and having a conversation about how it could happen, the strengths and weaknesses of the measurements, the internal challenges you will face.

When taking on something like this, it’s usually best not to try to change the world all at once, but one step at a time. So, for example, looking at the overall “unmeasurable budget”, what is the largest line item? If it’s “newspaper”, that’s a place you probably have the largest leverage. Implement there first, keeping in mind that this single implementation might help you down the road. For example, getting a unique phone number and results tracking for newspaper may teach you a lot about how to get this done for other marketing devices.

The finance people should be able to provide you with some idea of the net “margin” of a course and any other financial ideas that come into play. Then it’s a matter of asking if the spend on the media generated positive results. If the margin on a course is $500, a newspaper ad costing $1000 that only generates 1 student is not a great investment – but it might be the best one relative to other vehicles. This part is not really your call. Your job is to bring the data to life so that people can understand what they are spending and what they are getting.

There could be plenty of reasons why “losing $500” on a newspaper ad is OK – there is “brand exposure”, for example. In this case, the brand exposure only costs $500 versus a perception that it costs $1000, if the student generated is included in the formula. This may be a very positive result of the measurement for many folks in the institution. That judgment is really for someone else to make. Now at least they are making it on a full set of facts as opposed to perceptions they have about cost.

Q: However, how long do you keep measuring enrollments…they may not enroll based on one campaign…might take a few hits before they actually become students.

A: Sure. What seems reasonable? Given an annual budget cycle, let’s say reasonable is 12 months. One benefit from your tracking is you will be able to probably put some numbers against this eventually. If you get calls to a brochure number 2 years after it was issued, then the number is 2 years for a brochure. Newspaper calls stop coming in at 4 weeks, it’s 4 weeks for newspaper.

Q: The other issue: the marketing folks only want to measure up to inquiries–what they have control over.  What’s the best way to only measure the return on that…it’s before a “sale” or “enrollment” even occurs, so the “profit” is not booked.

A: Well, sometimes you simply have to decide what is “best available”. You certainly can start by measuring inquiries, especially since it’s pretty clear in this case marketing lacks some control over key conversion elements – financial aid, student abilities, and so forth. Down the road, it’s possible that certain types of media generate lower quality inquiries with lower conversion rates. You will get there over time.

For now, you could apply the “average conversion” to any lead to get down to the financial part of the game. If all leads on average convert at 25%, then just use that.  Then when tracking gets a wider reach, try to drill into it more deeply. To do this, you’d have to get access to enrollment data, of course. But you don’t have to get “all the data, all the time”. You could do a sample of a couple of months and go through it by hand to match back to inquiries, if you have to. You certainly would not be the first to do something like this to pin down an issue – it happens all the time.

Whether you want to do something like a “by hand count” or see it as part of your job is really more of a personal choice.  You can certainly – and analysts often do – blame a lack of knowledge on system problems, politics, whatever.  Just can’t get the data.  For some people “don’t know” is not acceptable – they have to find the answer, whatever way they have to do it – even if it is by hand!

Q: Does your book give an education example? We’re not “selling” a product and “sales” deals with # of credits taken by students & price per credit plus funding we receive from the state gov’t based on the number of FTEs generated. It seems like such a different animal so I’m struggling to figure how to do ROI for an educational services provider. I have created an Excel template based on how I calculated ROI for other industries but haven’t tried it yet.

A: Sounds like the “profit” side of it is a bit complex with the outside funding, but I’m sure you can get to a “value per FTE” somehow. Just start with something, and make it better as you move along. If you have to, simply take “all revenue” divided by number of FTE’s and you at least have a place to start.

The book doesn’t have extensive educational examples but here are some related topics from the newsletter:

Predicting Student Churn

Profiling Library Customers

I have a lot of interest in these educational measurement scenarios for a variety of reasons.  Keep me posted on how you are getting along and ask any questions you might have as you make your way through?  If I can be of help let me know!

Jim

Get the book at Booklocker.com

Find Out Specifically What is in the Book

Learn Customer Marketing Concepts and Metrics (site article list)

Download the first 9 chapters of the Drilling Down book: PDF 

6033% ROI, Defining Churn

DataBase Marketing, Driller Q & A, Newsletters

Jim answers questions from fellow Drillers
(More questions with answers here, Work Overview here, Index of concepts here)

6033% ROI
—————-

Q: We have exchanged email a few times, and I don’t recall if I ever said thank you for your book.  While I had been experimenting with many CRM programs in my little dry cleaning shop, your book gave my thoughts order and clarity to refine what I had started.  Today, I see the world differently.

A: Well, thanks for the thanks!

Q: You may or may not remember me. Just after I sold my dry cleaning shop, I had bought your Drilling Down book. I was the dry cleaner who had been doing rudimentary data mining and CRM with a point of sale system I had developed in Regina, Saskatchewan.

A: I do remember. Internally, I was thinking, “Wow, this is going to be a real test of the Drilling Down concept”? I mean, I have seen it work in many small businesses, but dry clean (seems to me) is a very tough, tough business. Too many players, a lot of competing on price, etc. A great environment for underground customer marketing in terms of beating the other guy – they will never know what happened to them. But still, tough for small owner / operator to have the “will” and time to really make it happen. So yea, I remember…

Q: Well, I’ve continued working within the dry cleaning as a marketing consultant. The programs I had developed in my shop have now been transplanted into a few of my client’s shops, and are bearing fruit.

Tonight one of my clients reported a ROI of 6033% doing direct mail to certain customers in his market in California. Another client of mine reported his fourth year of steady growth. One of my first clients has been showing a 7 percent annual compound growth, and he is in a flat or declining market. What began in my shop has been proven across North America, into Europe and Australia by my clients.

A: I can’t express how exciting that is. Congratulations!

Q: Jim, data mining dry cleaner’s data is a blast. You would be stunned at the quantity, and quality of data a dry cleaner gathers today. Would you ever have thought data mining could be applied to suits and shirts? Well yes, it can.

A: I am stunned, and I bow to your most excellent Drilling!

Q: Once again, thank you.

A: And thank you for sharing this, it’s very, very exciting to hear. Like you said, no other word for it than “stunning”. I remain most stunned!  Keep me informed. Perhaps you should write a book?

Jim

Defining Churn
———————

Q: I work for an economics consulting firm based in Washington DC. I am researching customer churn and customer displacement statistics across a variety of industries to try to establish a benchmark of what is considered high and low customer displacement.

A: Nice to meet you, and a noble task!

Q: Do you happen to have any such churn statistics, or know if a place you could recommend?  I found plenty of statistics regarding churn rates within the telecom industry, but am most interested in companies that are involved in business-to-business relationships with their customers (relationship between a customer and a supplier).

In addition, I would also like to find churn statistics for customers who use multiple suppliers. For example, a customer may go to several grocery stores rather than sticking with one dedicated store.  I would be interested in learning more about the statistics companies in these types of industries use to track customer displacement.

A: The reason you find a lot of churn info in telco / cable is the end of the customer life is easily defined by the disconnect, and these numbers are reported publicly as part of annual reports and so forth. In many other businesses like the ones you describe, typically the companies have failed to define customer defection and so in their minds, there is no churn because there is no defection.

A “customer”, even though they have not contacted the company for 3, 5 or 10 years, is always still a customer. If the company thinks like this there is no churn rate to be measured, by the definition the company has chosen for itself.

At the same time, defining defection is pretty easy to do by looking at the transactional data and defining the patterns of defection, for example “if a customer has not ordered from us in 3 years they are highly unlikely to order again”. That’s defection defined; you just put a line in the sand and say “3 years no contact is a defection”. The company then should declare customers in this status “defected” and then a churn rate could be found. This is pretty easy to do, so if not executed, one of two situations exist: either the company does not have the data or they don’t have the “will” to discuss, internally or externally, the concept of customer defection.

A third possibility exists: the company in fact has the data and has defined defection, but would never, ever speak to churn or customer defection in any kind of public forum because this information is so critically important from a competitive and strategy perspective. To discuss these numbers or the implications in public could have dramatic consequences for company positioning in the market or stock price. So if they have the numbers, they’re locked in a safe.

As a result, I’m sorry to say, I do not have any broad-based “sources” for you, save one possibility: a book called The Loyalty Effect by Frederick F. Reichheld (1996). In this book, Reichheld goes through the business models of 25 different companies that excel at retaining customers in different industries , and proves out the financial model of customer retention using real data. This is the book where the quote, “It costs 5x more to acquire a new customer than retain a current customer” (or the various bastardizations) came from. So it might help you out.

The only other thing I can suggest is that “churn” is not always the word used to describe these stats but is most often used when the disconnect is easily defined, as in telco / cable; “displacement” is a rare use for this idea as far as I can tell..

“Customer Turnover” is a popular phrase in Europe and is used by some in the US; also “defection rate” is used quite a bit. So if you’re pounding on Google to try to find these numbers, try those phrases and others you may find when doing these searches. Banking / finance / insurance is another area where the “disconnect” is often easily defined, so you will find various defection rates in some of their case studies on the web.

Jim

Get the book at Booklocker.com

Find Out Specifically What is in the Book

Learn Customer Marketing Concepts and Metrics (site article list)

Download the first 9 chapters of the Drilling Down book: PDF