Conversion / Testing

Lies and Statistics in Landing Page Optimization Tim Ash , CEO, SiteTuners.com - Jun 21, 2011

"There are three kinds of numbers: lies, damned lies, and statistics." -- Mark Twain, quoting Benjamin Disraeli

The statistics branch of mathematics has a poor reputation among the public. Much of modern science and economics is based on it in a fundamental way. So is public policy. Since public policy is a matter of priorities and heated debate about the allocation of government budgets, statistics has gotten pulled into the fray to support or undermine various political positions. Unscrupulous or ignorant people have corrupted it for their own purposes.

While there is nothing wrong with statistics itself, there are many common misuses of it. In this column, I have surveyed these along with some implications for landing page optimization.

Throwing Away Part of the Data

Statistical studies are based on a confidence level in the answer (commonly 95%). If you conduct a large number of experiments, even two identical effects can seem different based simply on a statistical streak. For example, if you flipped a coin five times you might be surprised to see it come up heads every time, and might even suspect that it could be loaded. However, this is exactly the result that we would expect based simply on random chance about 3% of the time. So if this experiment was repeated one hundred times, a series of all-heads would be expected to come up about three times.

Unscrupulous people might rerun the experiment many times, and report a single all-heads result as proof that the coin was loaded. By discarding the remaining experiments that did not support their desired conclusion, they are misrepresenting the results.

Traffic Filtering

In landing page testing, you generally want to get as wide a range of traffic sources as possible. That way, they are more likely to be representative of your visitor population as a whole. You generally want traffic sources that are recurring, controllable, and stable. If your traffic does not have these characteristics, it may be very hard to tune. For this reason, you may want to remove unstable sources (such as some of your larger but highly variable affiliates) from your testing mix. You should also generally remove nonrecurring e-mail traffic that arrives in spiky and sporadic "drops."

Sequential Testing

Another type of sampling bias can be introduced by sequential testing. For example, you may test your original design for a month, and then replace it with another one during the following month. It is hard to reach any kind of conclusions after this kind of experiment. Any number of external factors may have changed between the two testing periods. For example, there may have been a holiday with common family vacations, some major breaking news affected your industry, or you made a major public relations announcement. The point is, you are comparing apples to oranges. In landing page testing you should always try to collect data from your original version and your tested alternatives in parallel. This will allow you to control for (or at least detect and factor in) any changes in the external environment. Only use sequential testing as a last resort.

Short Data Collection

Even if you run your tests by splitting the available traffic and showing different versions of your site design in parallel, you may still run into biased sampling issues related to short data collection periods. Experiments involving very high data collection data rates may be especially prone to this.

For example, let's assume that you are testing two alternative versions of your page and are measuring click-throughs to a particular target page as your conversion action. Because of the high traffic to your landing page, you collect about 10,000 conversion actions in the first hour of your test. This data shows you that one of your versions outperforms the other to a very high level of statistical confidence. Many people would conclude the test at this point and immediately install the best performer as the new landing page.

But what if I were to tell you that the data was collected in the middle of the night? You might correctly conclude that people visiting your site during the day are a different population, or at least that they behave differently then. The same is true of weekday (accessing the Internet from work) versus weekend traffic (accessing the Internet from home). I suggest that regardless of your data rate you collect data for at least a one-week period (or multiple whole-week increments if your data rate is low). This will allow you to get rid of the short-term biases discussed earlier. Of course, this still does not address the question of longer-term seasonality.

Overgeneralization

Overgeneralization is the erroneous extension of your test conclusions to a setting where the original results no longer apply. For example, let's say that I set up an experiment to count the ants in my kitchen and tracked it for a full week during a record cold spell in the wintertime. My finding was that there were no ants in my kitchen at all during the study period. However, it would probably be incorrect to assume that the same would hold true during a heat wave in the summer. Often the overgeneralization is not made by the original researcher, but rather by those who subsequently summarize or cite the results.

A common overgeneralization in landing page testing is to assume that traffic sources that were not part of your original test will behave in the same way as the tested population. For example, if you see a particular effect with your PPC traffic, you should not assume that it will hold up when you expose the new landing page to your in-house e-mail list.

Avoid these common issues with improper use of statistics and you will be much more likely to find real conversion improvements in your landing page tests.

Tim Ash is the CEO of SiteTuners.com, a landing page optimization firm that offers conversion consulting, full-service, guaranteed-improvement tests, and software tools to improve conversion. SiteTuners' unique AttentionWizard.com visual attention simulation software tool can be used on a landing page screenshot or mock-up to quickly identify major conversion issues. Tim has worked with many leading companies and is a highly-regarded speaker at Internet marketing conferences. He is a contributing columnist to several industry publications and is the author of the bestselling book Landing Page Optimization.