What to Look for in A/B Split-Testing and Multivariate Testing Software for Small Business Websites
Web Marketing Today Premium, Issue 80, June 11, 2004
 You can purchase your own copy of this e-book for only US $11.99 or subscribe to Web Marketing Today Premium ($49.95 annually) to get this and about $160 of other e-books free. |
Summary: This article explains how testing software works, how cookies are used to identify visitors, and the two major types of software: page rotation and page element rotation. It discusses the pros and cons of CGI software vs. ASP hosted software applications and recommends a couple of applications of each type. Finally, it discusses the importance of various features in the software to look for when you are shopping for a product: multiple options, multiple campaigns, rotation weighting, action tracking, online reporting, graphic reports, revenue reports, statistical significance, SSL security, remote domain tracking, repeat visits, staged campaign closing, and multivariate testing.
Chapters in this series:
- Introduction to A/B Split-Testing for Your Webpages
- How to Test Your Landing Pages to Improve Conversion Rates
- What to Look for in Testing Software for Small Business Websites
- A/B Split and Multivariate Testing Software Reviews (13 products reviewed)
This article contains older information. Go here for newer information on conversion and testing.
There is an increasing number of A/B split-run testing software options available these days. This type of software is designed to test two or more options scientifically to see which option gets the best results -- sales, subscriptions, leads, or your other measure of "success". At the basic level, as visitors arrive at your test URL, the testing program shows them, on an equal basis, the various test options you have set up.
Cookies
When the visitor is redirected or "rotated" to one of the test pages, a cookie that identifies which of the test pages the visitor has seen is placed on the visitor's web browser. So the visitor will now have an "A Cookie" or a "B Cookie," etc. These cookies have two roles:
- Cookies prevent a visitor from seeing a different test page later if he or she were to return to re-read the offer. You don't want the visitor seeing two prices for the same product.
- Cookies identify which test page is responsible for a completed action when the visitor makes a purchase, subscribes to a newsletter, generates a lead, etc. Typically the "Thank you" page has an invisible GIF or some JavaScript code that reads the cookie and credits the completed action to the respective test page.
Types of Functionality
I classify the testing functionality I've seen into two groups:
- Page rotation. The various test options are complete separate webpages to which the visitors are redirected.
- Page element rotation. The test options are pulled dynamically into the page being tested by either JavaScript or Server Side Includes (SSIs) so the visitor is not redirected to another page.
Page Rotation
While page rotation is easier to accomplish from a programming point of view, it can cause serious problems if the page you are testing is your home page or a page that ranks high in the search engines. Typically, the program instructs you to employ a Meta Refresh tag that sends the visitor to the testing program for assignment, and then on to the test page option it has been assigned. Search engines are gun-shy of redirects, since spammers have used them to trick search engines. So when a search engine finds itself being redirected, it will change the URL in its index from the original URL landingpage.htm to the test option it finds, such as landingpage2.htm.
Here's what the Meta Refresh tag looks like:
<meta
http-equiv="Refresh"
content="0;URL=http://www.wilsonweb.info/sima/click.php?id=3">
When a browser sees this Refresh Metatag in the <HEAD> area of the webpage, it immediately sends the visitor’s web browser to the URL indicated in the tag. In this example above, it waits for "0" seconds (that's the number before the term URL in the tag) and redirects the visitor to my copy of Duncan Carver's Scientific Internet Marketing Assistant (SIMA) on my site, designated for test campaign 3 (id=3). Other programs may work somewhat differently, but this kind of redirect approach is common.
Page Element Rotation
Page element rotation, on the other hand, doesn't redirect the visitor to another URL. Instead, the optional content is dynamically inserted into an existing webpage, either using SSIs or JavaScript.
The advantage of this approach is two-fold:
- Search engines don't get confused and change the URL in their index to one of your temporary test option pages that will disappear once the test is over.
- You can potentially test several elements of your webpage at the same time. Be careful here, "don't try this at home." But if you are careful to set up a careful multivariate test, you can do your testing more rapidly. However, most low-end software doesn't support multivariate testing.
Hosting of Software
The other categories of testing software are:
- CGI software where you pay a one-time fee for a software license that permits you to download a program and install it on your own server. In general, these seem to be the simpler, less powerful products. Price for these varies from free to about $75, with several at about $30.
- ASP software that is hosted on the vendor's server. You are charged a monthly fee for this service. The hosted applications tend to be the more sophisticated, more powerful products.
If you're just getting started, you can purchase an adequate CGI testing program for a low price. Installation is pretty simple if you've had some experience. Or you can pay your hosting service a modest fee to install the program for you. Once it's set up, you'll find it easy to run all sorts of A/B splits and learn a great deal. Examples: Duncan Carver's Scientific Internet Marketing Assistant(SIMA; free) and Phil Huff's Split Test Generator 2. If you need support and coaching, consider Marty Foley's Scientific Web Marketing System which comes with his ConvertMoreTraffic monthly subscription.
However, if you are part of a larger company with a good bit of traffic to your site and a reasonable marketing budget, you'll want to consider a hosted ASP application. You'll get more power and support with the hosted software route, and the software is usually kept up-to-date. Examples: Vertster (lower cost) and Offermatica (moderate cost). Both of these offer 30-day free trials.
Some Things to Look For
When you're shopping for a software program, here are some of the features you should be considering. Not all are absolutely necessary, but some are important to make the process go easier.
Multiple Options. A few programs offer only two test options at a time, an A and a B option. You need to look for something that offers you at least 5 options. Fortunately, most do.
Multiple Campaigns. Look for a program that allows you to set up multiple campaigns so you can have several tests going on at the same time. For example, I'm currently testing (1) a newsletter subscription pop-up, (2) a page that encourages subscribers to confirm their subscription, and (3) the landing page for an e-book.
Rotation Weighting. The lower end programs will divide the traffic equally between the number of test options you offer. The better ones allow you to weight the traffic however you choose, such as 60% to your control page (A), 20% to B, and 20% to C. This way you aren't risking a major amount of income if the test options don't convert well. This feature is probably only important for sites with a lot of traffic. Most smaller traffic sites will be hard-pressed to get enough traffic and click-throughs to achieve a statistically valid test result.
Action Tracking. Most programs not only distribute visitors evenly to your test pages, they also keep track of those that "convert" or click-through to take the desired action. This is a must! Otherwise you will need to gather your conversion data from one source and put it together with data on pageviews in order to get conversion rates of each of your options. You don't need more work.
Online Reporting. Nearly all the programs have a real-time read-out of the current status of your various campaigns. At a minimum, the report should show for each test option the number of page views, the number of click-throughs, and the conversion rate. The program also ought to have a place where you can record notes on how the test was set up.
Graphic Reports. If the reporting mechanism shows graphs, all the better. They can help you tell at a glance what you need to know and how significant it might be.
Revenue Reports. It's not necessary to display on the reporting page the amount of revenue generated for each test option, but it certainly helps you get a clear picture of the economic impact of various test options. Some programs I've seen allow you to enter the dollar amount of the product when you set up the test. But a more accurate approach would be to program the thank you page to extract the subtotal and transfer it to the tracking program when a sale is made. This is helpful when you are testing one price against another.
Statistical Significance. Testing programs should provide is an assessment of the statistical validity of the data you are seeing as an integral part of the report. So far I haven't seen any program that includes this. The statistics needed are (1) confidence level and (2) margin of error (why not, since all this is a matter of programming in a mathematical formula) -- along with a plain English explanation of what each of these means with regard to the data so far collected for your campaign. Many people using these programs are statistical newbies, so give them a break!
SSL Security. A program ought to include SSL security so that the cookie read from a secure thank you page of the ordering system won't show a warning that says, "Beware, not all the images on this page are secure." If you set up a program on your site, you'll need a digital certificate on your site so that the invisible GIF on the thank you page can begin with either http://... or https://....
Remote Domains. Most of these programs can track a page from a domain different than the one on which the tracking program is hosted. This is desirable if you are handling several sites or have your e-commerce function on a separate site than your own. Programs that use SSIs will need the tracking program hosted on the same site.
Repeat Visits. For some products, people will come back again and again to the landing page where the offer is made before they finally decide to make the purchase. Some programs will tell you on which visit the purchase was made.
Staged Campaign Closing. With lower-end software, you close a test by stopping the automatic rotation to the test options. But what about the visitors who take several weeks to make up their minds? Ken Evoy, author of Make Your Site Sell! and developer of Site Build It! told me of the time he was testing a short sales letter against a webpage with a sales letter so long that it "went out the door and across the street." Initially, the short sales letter won overwhelmingly. Ken closed the test to new visitors, but kept the test open to record sales by visitors who had already viewed one of the sales letters. As they gradually converted, the long sales latter finally equaled the response of the short sales letter. Ken said he wouldn't have known this if he hadn't had the capability to close the test to new visitors while keeping the sales tracking open. I don't know of any programs that allow this type of staged campaign closing at present.
Multivariate Testing. As mentioned above, most programs will allow you to test various options for a single element of a landing page. But some of the higher end programs, such as Offermatica and Optimost, allow you to test several elements of the landing page at the same time using sophisticated matrix testing, hill climbing, or Taguchi Method multivariate testing approaches. Most smaller traffic sites just don't have enough traffic to warrant multivariate testing. But high traffic sites may find this an excellent tool to rapidly optimize a low-performing site. After optimizing a site, however, it is important to continue to use standard A/B split-run testing on a regular basis to achieve continuous incremental improvement and deal with inevitable changes in the selling environment.
Chapters in this series:
- Introduction to A/B Split-Testing for Your Webpages
- How to Test Your Landing Pages to Improve Conversion Rates
- What to Look for in Testing Software for Small Business Websites
- A/B Split and Multivariate Testing Software Reviews (13 products reviewed)
Sample newsletter. We respect your privacy and never sell or rent our subscriber lists. Subscribing will not result in more spam! I guarantee it!