How to Test Your Landing Pages to Improve Conversion Rates: A/B Split-Testing and Multivariate Testing

by Dr. Ralph F. Wilson, E-Commerce Consultant
Web Marketing Today Premium, Issue 80, June 11, 2004

You can purchase your own copy of this e-book for only US $11.99 or subscribe to Web Marketing Today Premium ($49.95 annually) to get this and about $160 of other e-books free.

Summary: This article discusses the theory behind A/B Split-Testing, so you know you are "comparing apples with apples," and getting an accurate test. It goes over determining statistical validity of tests, confidence levels, percent error, etc. Next it outlines the kinds of elements of a landing page to test, the elements that are likely to make the most difference. One section examines multivariate testing, sometimes called matrix testing or Taguchi Method for design of experiments, that tests several elements at the same time. Finally, it outlines a 5-step program for small businesses to get started in scientific website testing.

Chapters in this series:

1. Introduction to A/B Split-Testing for Your Webpages
2. How to Test Your Landing Pages to Improve Conversion Rates
3. What to Look for in Testing Software for Small Business Websites
4. A/B Split and Multivariate Testing Software Reviews (13 products reviewed)

---

This article contains older information. Go here for newer information on conversion and testing.

For a while you can get away with guessing, shooting in the dark, and extrapolating from number of orders, pageviews, etc. But at some point, if you want to improve your sales on the Internet, you need to get serious about testing scientifically. Fortunately, it's much easier than you think and quite inexpensive -- at least the way I'll show you. I don't know why I didn't start split-testing sooner, except that I thought it was too difficult. I was wrong.

A/B Split-Testing -- Comparing Apples with Apples

Web marketers do a lot of informal tests. You can calculate the conversion rate for a subscription form or a landing page. You make a change and the conversion rate goes up or down -- or does it? A sequential test can be influenced by day of the week, current events, and lots of other things.

The only accurate way to test is to send equal numbers of visitors to different permutations or versions of the same web page. In the direct marketing industry this is called A/B split-run testing. In its simplest form, a testing program alternately sends visitors in equal numbers to see different versions of the webpage:

Visitor 1	sees	webpage version A	and gets	an A cookie
Visitor 2	"	webpage version B	"	a B cookie
Visitor 3	"	webpage version A	"	an A cookie
Visitor 4	"	webpage version B	"	a B cookie

and so on until the test is done.

So long as you have adequate traffic, the time of day and day of the week will be exactly the same for both test groups of visitors, group A and group B. When you track the actions of group A and group B visitors, including a count of the sales or subscriptions or leads generated by each group, you can tell what effect, if any, your changes made on performance. Typically, on the "thank you" page after a sale or subscription, the program will read the cookie of the visitor, determine whether it was an A cookie or a B cookie, and record this information in the testing program's database. Here's what it might look like at the end of a test:

Group	Pageviews	Click-throughs or sales	Conversion rate
A	8675	325	3.75%
B	8675	155	1.79%

Clearly, version A of the webpage got a better conversion rate in this study -- about twice as good!

But there's no reason to stop at only two versions or options, just A and B. Most split-testing tools let you test up to five or more versions. This morning I set up a test on the landing page for one of my e-books this way. The control page is the way the webpage was when I started the test and allows me to see how well other changes perform.

A	Landing page with no sound (control).
B	Adds sound as an option for the visitor.
C	Adds sound that plays automatically when someone visits the webpage.

It'll probably take several weeks or months to get enough data for a valid conclusion, but this is how I start.

Statistical Validity

How valid is this kind of testing? The more visitors you have and more sales you make, the more accurate the test, of course. But there are ways to be much more precise. Statisticians and direct marketers have a complex formula to calculate this. They use terms like:

• Confidence Level -- the chances that you'll get a repeatable result. A 95% confidence level means that you'll get a repeatable result on 95% of subsequent similar size samples. In other words, there's a 95% chance that these test results aren't just a fluke.
• Limits of Error -- the expected variation in the response or conversion rate from one sample to another. A 0.2% limit of error for a 1% conversion rate means that you'd get between 1.2% and 0.8% conversion rate.

The best tool I've found to quickly compute the statistical value of your test conclusions is Konstantin Goudkov's free online split-run test calculator that does the math silently and gives you a meaningful answer in plain English, though the free version is limited to smaller test samples. http://calc.in-the-name-of-profit.com/86429/ab_test.jsp Another tool to consider is Vertster's Click-Through Rate Validity Checker. It was designed for use with Pay Per Click advertising campaigns, but it can just as easily be used to check the validity of split-testing results. http://www.vertster.com/adwords-tool/ Also consider Perry S. Marshall's free SplitTester.com tool. It isn't as easy to understand, but will take larger sample sizes. http://www.splittester.com

So just how big should your sample size be? Short of doing the math, marketers tend to shuffle their feet and cough ever so slightly. They mutter, "It all depends." Vertster's Scott Miller offers a rule of thumb that you should keep testing until each offer has at least 50 click-throughs or orders and until the conversion rate doesn't change much with each new click-through or order.

Mike Sack, executive VP of Inceptor, is quoted in ClickZ as advising: "To be sure your test is showing a statistically meaningful impact on the variables, you have to know if you've demonstrated enough of a difference (delta) between the tests to declare a clear winner. As a rule of thumb, you should have at least a three times larger result (e.g., if A is 5, B should be 15)." Of course, if you have enough pageviews and click-throughs, you can be sure of your data with a smaller delta, but that may take time.

Conversion Expert Bryan Eisenberg of GrokDotCom.com suggests setting up an A/B/A test, that is, two A versions and a single B version. "When the results from the two A options begin to converge," he says, "you know your getting to the point that the B option is also reliable."

The wise answer for small businesses is to keep your tests running, see if the conversion rates are stabilizing, and don't stop them prematurely. However, A/B split-run testing is most useful on sites where you have enough traffic and sales to collect meaningful data in a few days or weeks. Enough visitors and buyers is one reason that serious online retailers for major brands are now getting fairly high conversion rates. They've had both the traffic and budget to tweak their websites over a significant period of time.

What to Test

You could test everything on your landing page, but that would be a waste of time. According to Bryan Eisenberg, the most important items to test on landing pages include:

• Bonus gifts
• Coupons
• P.S. messages
• Guarantees
• Opening sentence image
• Closing sentence image
• Calls to action
• Headings
• Colors
• Locations of elements
• Hyperlinks

When you're testing ten elements, one after another, it can be a daunting task. Isn't there a simpler way? Yes, if you have enough traffic for meaningful data and enough sophistication to pull it off.

Multivariate Testing

Multivariate testing (pronounced mul-ti-VAIR-ee-ate) means testing multiple variables, sometimes called matrix testing, "hill climbing," or Taguchi Method Orthogonal Arrays. They're just beginning to be used in testing webpages and landing pages, but they've been used for 25 years in industry.

Dr. Genichi Taguchi -- a pioneer in the Total Quality Management movement (sometimes known as 6 Sigma) that revolutionized Japanese manufacturing, and later, US manufacturing -- devised a method to cut down on the number of tests needed to gather the necessary data to design a more productive manufacturing process. He discovered which design factors were important by setting up an orthogonal array. I'll spare you the details. Suffice it to say that while this method can speed up testing of a new website design, for example, on occasion it can mask problems that can only be identified by sequential split-testing. Nevertheless, multivariate testing can substantially speed up the process.

At present, I know of only two programs that support multivariate testing -- Offermatica and Optimost.

What Is a Small Business to Do?

If your eyes are glazed over by now, I understand. But I didn't want to insult you by trying to dumb-down this information too much. You need to know what are the strengths and danger points of various types of testing.

If you are a small business, I recommend that you forget multivariate testing at the present and concentrate on doing simpler A/B Split-run Testing. Here are the steps:

1. Split-testing Program. Install one of the inexpensive CGI programs or sign up for one of the ASP hosted programs that I discuss in my reviews of split-testing software.
2. Set up a test of your most critical landing page problem. For me this was increasing the subscription rate for my newsletter. For you it may be making a landing page work. NOTE: Be careful of testing pages where you search engine ranking is vital, since many programs use redirects to the test pages that can confuse search engine indexing.
3. Let the test run. Don't stop until you've accumulated enough data to see a clear, statistically significant difference between the options. But while you're waiting, you'll probably be able to explore and test some other factors with additional experiments. Don't stop.
4. Make the changes indicated and test other factors. All this time you'll be watching your conversion rate inch up -- perhaps farther than you'd ever dared hope. Many national online retailers are reporting 5% to 10% conversion rates, and sometimes higher.
5. Keep a journal of your results. It's easy to lose or forget some of the golden data you'll uncover. Save it all in an online journal or a bound journal near your desk. Going back over some of your previous tests may give you clues to a problem you'll be struggling with six months from now.

If you're not doing A/B split-testing now, you have a big upside -- a lot of sales and revenue to gain. Get started now!

---

Chapters in this series:

1. Introduction to A/B Split-Testing for Your Webpages
2. How to Test Your Landing Pages to Improve Conversion Rates
3. What to Look for in Testing Software for Small Business Websites
4. A/B Split and Multivariate Testing Software Reviews (13 products reviewed)

---

Other articles from Web Marketing Today Premium, Issue 80, June 11, 2004