Conversion / Testing

Email this article Printer friendly page

With a split-test or two under your belt, you may start wondering how long your test results will remain valid. The fact remains that if you run a split test of a certain element and get one result and repeat the test a year later, you may not get the same result.

Remember, a winning test result is an indication of consumer preference. Running a split test between button styles tells you that during the test, your users preferred one style over the others. Likewise with a benefit headline: your testing will show that consumer preference indicates that one benefit is the most important in the buying decision.

As marketers, we know that consumer preference is fickle and can be highly variable. To better understand the longevity and reliability of your results, let's review a few of the more common causes that might invalidate your test results over time.

1. The constantly evolving market.  Now more than ever, our markets seem to be evolving at a rapid pace. Particularly with Internet or technology businesses, what causes your product to sell today probably will not work tomorrow. Disruptive new technologies, business models, and services appear every day that change consumer preferences.
2. Internet users are getting smarter. Every day that passes, the online population gains more experience at navigating the web. This can have a large effect on usability items such as buttons, forms, and complex systems. The easy, dumbed-down version of your page may have resulted in increased conversions last year. But now that people are smarter, that dumbed-down version may fare worse because it is slower than the "bare bones expert version" and creates a roadblock for today's savvier user. 
3. The competitive landscape is always changing. Your company is always changing, and so is your competition. If your leading competitor launches a new ad campaign that your current marketing message does not address, you may be forced to change your messaging.
4. Seasonal Effects. One of the most predictable sources of variation in consumer preference is seasonality. Although we often marginalize this as "noise" in our tests, for some products it can have a very profound effect. Consider a swimwear maker. In the summer, a headline split test might yield, "How to look great on the beach tonight, tomorrow, and this weekend," as the easy winner. However, fast forward six months and in most of North America, this headline would make no sense at all, because no one goes to the beach in the middle of winter.

Primary selling benefits that change frequently, such as those which are highly influenced by seasonality, should be re-tested as often as every month or two. Usability tests enjoy a much longer shelf life, sometimes greater than a year or two. Unfortunately there is no hard rule.

The best approach is to continue testing, all the time. Use a web analytics solution to understand your baseline conversion rate, and keep a close eye on any changes. Re-testing key pages and offers on an annual basis is a good idea.

Scott Miller is the CEO of Vertster.com and the author of the book, The ConversionLab.com: How to Experiment Your Way to Increased Web Sales Using Split Test and Taguchi Optimization. This past year he has overseen more than 2,500 web experiments.