We’re big fans of hack analysis when clean tests aren’t possible. I may have to quote you on the notion that just because we can’t see every causal connection doesn’t mean we shouldn’t try. Too many marketers simply throw up their hands and spend more money.

Though far to few, I expect!

OK, with the ass-kissing behind us…

There is one thing you mention that I would take issue with. You wrote: “The job of an analyst is to seek the truth – an ongoing process, with “better truths” exposed as you move forward.”

I would submit that the job of an analyst is to improve business performance. As a business exec, I’ll leave the question of “truth” to the philosophers. I want an analyst to help me improve profits.

http://www.jimnovo.com/newsletter-2-2006.htm#Q1

If you repeatedly get the same result when you take out the advertising, it’s difficult to dispute there is a hard linkage. You may never know how to specifically describe that linkage between the ads and goal behavior, but you don’t have to understand the linkage to determine it exists.

This curiosity about “linkage” is in fact the basis of much scientific experimentation – I can see the linkage, but why does it exist?

For example, I see that when flies are on meat, people tend to get sick when they eat the meat.

Proving why these linkages exist are the basic work of the scientific method. Fortunately for most Marketing applications, proving the specific mechanics of these linkages is not nearly as important as proving they exist in the first place, determining with reasonable clarity there is cause and effect.

I can cover the meat and get real benefits without knowing for a century why, specifically, covering the meat prevents disease.

Many #wa folks would often be better off just going ahead and “covering the meat” rather than letting the flies gather while they figure out the precise mechanism that leads to disease.

If ya know what mean…

Thank you, I think that was the lightbulb I needed. I specifically remember one time we paused PPC advertising for a month for a smaller client and there was a noticeable decline the in leads generated.

I appreciate the help, thanks again :-)

Control = no advertising. Sometimes in a micro market like this, the only way to create control is to stop the advertising, then compare results when ads are running versus not running. You could do this one publication at a time, if desired.

Critics will cry this is not a true control, since different time periods are used and “something” migh happen in one period that did not happen in the other to affect results. True, from a statistical perspective.

But clearly, the analytics Ninja could monitor traffic and look for the presence of outlier events, eventually declaring the two time periods to be equal *except* for the change in advertising. If outlier events did occur, the traffic from these events could be excluded from the analysis.

While “on/off” testing like this is not perfect from a stats perspective, it’s certainly better than not testing at all, and if the results can be repeated 3 times, you likely have “Marketing Significance”.

If you were testing a new drug for efficacy or determining defect rates on parts for military aircraft, I would agree the above method for creating control is not statistically valid. But we’re talking about Marketing here, and the search for better truths using limited resources.

Here’s my particular situation: our firm is small and specializes strictly in luxury real estate, mostly on the east coast (we have accounts in Costa Rica, Atlanta, New Jersey, but the majority are in NYC).

So, by definition, we have a small amount of people who visit our websites. On top of that, the number of leads we generate is fairly smaller. So there’s the first problem: our data set (to me) seems teeny-tiny. Next, I would think our data is difficult to separate out into control groups. A lot of our advertising is hyper-local, such as banners on websites that only people in NYC would know about. And, I would argue, most the visits to the website are because of our marketing efforts. Finally, we only use Google Analytics on our sites and I think the most I can zoom in is by city.

But, as Avinash has taught me, there is a purpose to our website, and so (almost) every metric we look at in Analytics is measured against one goal: generating leads. And now, as you have instructed me, I need to figure out “lift” in our campaigns. I feel like I’ve struck gold with this idea… I’ve had financial guys try to calculate ROI in front of me at meetings and rail against us if they think their investment is not getting their money’s worth. Next, our clients are pulling back dollars and it would be wonderful to have hard data to say Publication A and Publication B are *proven* to “lift” leads so they don’t feel like a guinea pig.

So… a control group. I would love any thoughts on the situation. Thanks!

While it’s important to have the right data, clean data and all that, we can’t let these issues prevent us from discovering better truths. So what we do, if I can be permitted to use the term, is “hack” a solution. These hacks can be extremely creative, I’ve seen some wonderous stuff along these lines over time.

There’s not really enough info in your comment to propose a solution, but to generalize, in hyper-local situations I try to use zip codes or (even more hyper) street maps. Assuming you are selling something, do you have street address?

For example, pinpoint a location to stage a streeet promotion of some kind, say, simply a kiosk where you do product demo or simply display product and engage people passing by.

Look at net lift in customers for those streets versus the whole borough. In this case, the rest of the borough is “control” and “test” is the area, say a 1/2 mile radius, around where you do the street promo. You should see customer counts growing more rapidly (%) in the test area versus control.

This is a rough example, fine-tune for location and product. But it does give you an example of how powerful the test / control idea can be.

For another example of hyper-local promo using geo-demos, see:

Geo-Demos That Work

Same thing, test and control.