Are your marketing tests meaningful?

Can you remember back to the days in middle school or high school, when you would get a test, take one look, and realize, “all they want me to do is spit back what they already know?”  No analysis, no way to show that you had learned to apply the information — just like Joe Friday, “just the facts, ma’am.”

While Joe eventually would use the facts and catch the criminal in the end, the feeling I got from such fact-based tests was pure frustration.  Look, it is clear that you do need to know the facts; there is no denying that it is hard to understand history without a core knowledge of what happened and when.  Yet, that is just the beginning, not the final destination.  This is what I felt:

Asking me only to replay back what you already know wastes both of our time.

In the past week, I have been asked two times to evaluate test programs that used customer data to differentiate target, offers and communications.  Both programs were well thought out in terms of target — they were trying to use differences in segments as the basis for different offers.  All of that made sense, and was consistent with Best Practices in data-driven marketing.

Yet both tests ended up showing no meaningful difference at all between the “champion” and the “challenger” efforts.  For those who do not know, champion-challenger is a methodology where marketers test their best performing communications for a customer segment against an alternative.  Just like in “Iron Chef,” the winner stays and the loser is replaced by another test candidate.

In both of these tests, there was no significant difference between the results for the Champion offer/communications and the Challenger.  When I looked at the stream of communications in each group, the answer was obvious…

The reason there was no difference in results is because there was no real difference in the communications!

What they chose to test was so small in difference that consumers could not tell them apart.  As a result, the percentage of consumers responding showed no statistical difference.  A lot of resources (time, money, bandwidth) wasted, all for nothing.

To see a difference between approaches, there has to be one. How will you know?

• When you are testing, whether direct mail, email or telephone scripts, the differences have to be B-Squared:  BIG and BOLD.
• Not in the eyes of the company, but in the eyes of consumers, who are harried, busy and overwhelmed with their lives in general.
• If you are testing anything subtle, give it up (with the exception of some web site experience tests).  Don’t test a blue background vs. a red one, or a boy vs. a man, or 35 cents vs. 45 cents.

Test discounts vs. services vs. content.  Test one communication against five.  Test locally focused vs. brand focused.  But whatever you do, you must make it so BIG that no one misses it, and so BOLD that it will matter when read.  It actually doesn’t matter if the benefit is so big you could never roll it out across all consumers or all markets — do it anyway.

Then, once you have found a difference between one end of the spectrum and the other,  you can begin to test trimmed-down versions that can meet the company’s capabilities and appetite for difference approaches.  At least then, you will know how high you can jump before lowering the bar.

The worst result for any of us is “no difference.”  In work, as in life, we all want to know we make a difference.  Tests that are too small to read demotivate your team, and waste your precious time.

So go big, make a splash, and THEN tone it down.  At least that way, your work will never be forgotten.

Do you have a story of a test that was too timid to succeed?

Related articles by Zemanta

• 5 Common A/B & Multivariate Testing Mistakes (getelastic.com)
• Optimization Testing 101: Part 1 – Introduction to A/B and MVT (getelastic.com)
• New reporting features: test summary and cumulative charts (visualwebsiteoptimizer.com)
• A Complete Guide to A/B Testing (webdesignerdepot.com)

Tweet

Tagged as: customer segmentation, marketing measurement, marketing metrics, Testing