A/B Split and Multivariate Testing Software Reviews
Web Marketing Today Premium, Issue 80, June 11, 2004
 You can purchase your own copy of this e-book for only US $11.99 or subscribe to Web Marketing Today Premium ($49.95 annually) to get this and about $160 of other e-books free. |
Chapters in this series:
- Introduction to A/B Split-Testing for Your Webpages
- How to Test Your Landing Pages to Improve Conversion Rates
- What to Look for in Testing Software for Small Business Websites
- A/B Split and Multivariate Testing Software Reviews (13 products reviewed)
These days the market is offering an increasing number of A/B split run testing software programs. Here are the main ones currently available. In the chart below I have divided them according to type:
ASP Hosted Applications
CGI Programs
- Duncan Carver's Scientific Internet Marketing Assistant
- Phil Huff's Split Test Generator 2
- SplitHit Professional Split-Run Testing Software
CGI Programs using SSIs
JavaScript Rotation Generators
Ad Tracking Programs with a Split-Testing Feature
ASP Hosted Applications
Inceptor
Inceptor
http://www.inceptor.com
| US
Headquarters: 4 Clock Tower Place Suite 300 Maynard MA 01754 Phone: 978.298.1525 |
London Office: Highlander Plc 7th Floor Northern and Shell Tower London UK E14 9GL Phone: +44 870 330 9900 |
This program has very little explanation of its A/B testing programs on the Inceptor website, but it is a higher- end program that comes highly recommended by people I trust in this field.
Offermatica
Offermatica.com
http://www.fortpoint.com
Fort Point Partners, Inc.
Pier One Bay Three,
San Francisco, CA 94111
Phone (866) 627-3557
5˘ per visitor
If you have limited control over your website or need advanced features, here's a hosted ASP application that doesn't have to be hosted on your site at all. It uses JavaScript to call a .js application from the third party site. The program works differently than most of the others. You define one or more regions on your webpage or webpages as an mBox by pasting in a couple of lines of JavaScript for each mBox. Then you upload to Offermatica content and targeting rules that will be displayed where appropriate in the mBoxes you have created. Real-time reports and A/B test results are available online in graphic form. Pricing is on a 5˘ per visit basis, no matter how many active mBoxes you are tracking. Discounts for higher traffic sites are available, as is training.
The software allows you to target which visitors see which content. Targeting can be by ad source, new visitor, and segment. It can display automated product suggestions such as "top sellers," "most viewed," and random product rotation.
The program provides for simple A/B testing. But what is really interesting is its ability to test multiple factors on a webpage at the same time. Using an A/B testing approach to study 5 different factors on a page sequentially could take weeks or months. But with Offermatica's multivariate testing using the Taguchi method, you could study all 5 factors simultaneously. Call it "higher math" or whatever. Imagine what this could do for a website design firm seeking to quickly optimize a website or a landing page.
I'm very impressed with the capability of Offermatica. This is a relatively new but very sophisticated tool from a firm that has done website and e-commerce development for top online retailers. The tool, however, is designed to be affordable for both smaller and very large companies. Currently Offermatica is offering a 30-day free trial.
Optimost
Optimost
http://www.optimost.com
750 Third Avenue, 16th Floor
New York, NY 10017
Optimost is positioned as a high- end hosted ASP toolset that serves medium- to large-sized companies. They don't display prices on their website.
According to their site, you start with a creative you want to test -- an ad, webpage, landing page, etc. Then select several of the elements you want to test, such as the headline, images, layout, elements of the offer, the guarantee, etc. Now Optimost's system dynamically creates thousands of permutations of the creative, then sets up a sophisticated experimental design to run systematic, iterative tests. The reporting engine then evaluates which of the factors or elements you are testing significantly improves response rates. Optimost dynamically creates a new creative based on the knowledge it has just gained and then continues to run iterative tests to continually improve performance.
Optimist comes in two flavors -- Optimost Select for working with individual creatives or webpages and Optimost Enterprise for continuous development projects. This looks like a good approach for medium to large companies, though webpages constructed completely by machine-dictated variables seems like over-kill. Usually, only a few elements on a webpage make much difference in its performance. Only these really need testing.
Verster Conversion Rate Testing
Vertster Conversion Rate Testing
http://www.vertster.com
U-Pro, Inc.
Scott Miller
337 South 400 East, Suite 400
Salt Lake City, Utah 84111
(801) 363-6200
$17.99/month for up to 20,000 page views
$37.99/month for up to 50,00 page views
Vertster is the only hosted ASP application I've seen designed to help small businesses improve their conversion rates.. Vertster allows you to set the ratio of traffic that goes to a test page so that you can maintain your revenue stream while testing, such as 70% Control, 15% Test A and 15% Test B. Vertster tracks both page views on the test pages as well as click-throughs that result in sales or conversions. The program offers over 100 "boardroom quality" graphs and reports to help you understand the results you are seeing.
Vertster was originally designed for PPC marketers seeking to optimize their webpages. Though it currently has a few rough edges, this is the ASP solution for small business to watch. The program is under active development at present. The site also includes a number of articles explaining the ins and outs of A/B testing and a Click-Through Rate Validity Checker.
Visual Sciences
Visual
Sciences
http://www.visualsciences.com
1616 Anderson Road, McLean, VA 22102
Phone (202) 595-0066
Their website says: "Visual Sciences’ software applications help our clients understand and profit from their large volumes of customer activity data. The Fortune 500 companies that rely on us call us their metrics, analytics, statistical analysis, or reporting software vendor. Presently we offer Visual Site™ for Internet channels and Visual IVR™ for voice channels. Each of these applications provides our clients an integrated real-time system for collecting, processing, analyzing and visualizing their data. If your company operates in the financial services, travel, media, retail, telecom or high-tech industries, you need to see our software."
There are no prices -- or anything else on their website -- but I'm told by people I trust that they are a great company for testing.
CGI Programs
Duncan Carver's Scientific Internet Marketing Assistant
Duncan Carver's Scientific Internet Marketing Assistant
http://www.marketing-strategy.org/downloads/
Duncan Carver
Marketing-Strategy
Lower Hutt, New Zealand
This software is installed on your website and requires PHP 4.1.0+. This program offers some nice features and is free. It allows you to test an unlimited number of options -- though most folks will usually try two or three versions at a time. Like several split-testing programs, it can be used to do split-testing on other domains than the one on which it is installed. Though it uses PHP as its programming language, it doesn't require you to test only .php webpages -- regular .htm webpages work fine.
SIMA also uses an invisible GIF image to track the "thank you" page to determine how many desired actions (sales, subscriptions, leads) were completed and computes the conversion rate for each option. It does not track the dollar amount of sales.
This program has a couple of unique features. First, it keeps track of how many times a visitor looks at the test page before making a purchase. Second, you can decide how many visitors you want to test -- say 4,000. Third, at your choice the program will run the test for 4,000 visitors, then automatically select the best option of the webpages tested and redirect all traffic to that page in the future. When you get back to it, you can make the change permanent, but this keeps you from wasting improvements highlighted by your test should you become preoccupied elsewhere.
The entire program is written in PHP (without any MySQL connection, incidentally). I first tried to install it on a server that had PHP installed only as a CGI -- it didn't work. Then I installed the program on a server with a full installation of PHP as an Apache Module -- voila! It worked fine. Since this is a free program, support isn't available via phone or e-mail. However, the online forum discusses various problems and provides solutions.
On some of my tests so far, the program doesn't seem to be tracking nearly all the daily visitors that I know have clicked-through to subscribe to my newsletter. It's hard to know how to interpret this discrepancy.
SIMA tracks page views, click-throughs, and conversion rates for an unlimited number of options, can run multiple campaigns at one time, is configured via web browser, can do its magic on remote domains, and is free! A good choice so long as you can run PHP.
Phil Huff's Split Test Generator 2
Phil Huff's Split Test Generator 2
http://www.splittestgenerator.com
Phil Huff, Diversified Financial Group
PO Box 14, Mount Pleasant, SC 29465
(843) 884-9851
Price $29.97
This is a Perl script installed on your server. With it you can run multiple split-tests and track conversions on any shopping cart where you control content on the thank you page. It can test up to five different variations at one time. A free copy of the older version 1 (that doesn't calculate click-throughs) is available for subscribing to a newsletter.
The administration screen shows something like the following for each test you administer :
| URL |
Views |
Click |
Success Rate |
Total Value |
|
http://www.domain.com/product1/index1.htm |
3056 |
248 |
8.1% |
$3,720 |
|
http://www.domain.com/product1/index2.htm |
3054 |
471 |
15.4% |
$7,065 |
The total value of sales is derived by multiplying the click-throughs times the price of the product you set in the administration interface when you set up the test. It is not calculated from the actual sales total of your ordering system.
While this is a fairly simple program, I liked it for several reasons. (1) It is fairly easy to install; (2) it quickly displays the essential information that marketers want to know (i.e., relative conversion rates); (3) it allows set-up from a web browser interface, (4) it enables you to run several tests simultaneously, if you like, and (5) it comes with a very modest price tag.
SplitHit Professional Split-Run Testing Software
SplitHit Professional Split-Run Testing Software
http://www.wilsonweb.com/afd/splithit.htm
Jooz Software
Michael Wagner
A-5020 Salzburg,
Lasserstr. 27, Austria
$29
SplitHit is written in PHP with a MySQL database backend and is installed on your webserver. The administrative area allows you to add, modify and delete index pages, versions, and target pages. It also shows you the number of hits and repeat visitors for each version, the most popular paths taken, paths taken to the target URL, and visitors that reach the target URL. Reports are shown with bar graphs. Requires PHP 4+ and MySQL on a Unix server.
From looking at the relatively passive support forum, it seems that the company is in just in the process of releasing version 2, which will include ad tracking capabilities. A beta version seems to be available to those who purchase version 1. Apparently the program won't work easily unless PHP is installed as an Apache module rather than as a CGI.
CGI Programs using SSIs
Marty Foley's Scientific Web Marketing System
Marty
Foley's Scientific Web Marketing System
http://ProfitInfo.com/cgi-bin/v4.cgi?PI904
Marty Foley
ProfitInfo.com
1610 Crestdale #9,
Houston, TX 77080
Phone (713) 467-3933
One of the pioneer A/B split-testing scripts available for Web marketers, this software is now provided at no extra cost to members to Marty Foley's ConvertMoreTraffic website, $23.95/month or $247/year. This is a CGI Perl 5 script that installs on your own Unix server and uses SSIs. The system puts a cookie on the visitor's browser, so the visitor sees the same version of the webpage upon returning.
This program is installed by uploading the small script to your cgi-bin directory after changing some path locations to match your server. Unfortunately, page views and click-throughs for each option can't be monitored easily using a web browser. The program's log files can be activated to display how many hits went to each option, but they must be viewed by going to the log file via FTP, opening the file, and viewing the file. There is no built-in method to compute conversion rates and sales, since the system only handles displaying the content and doesn't read a cookie on the thank you page. The program only handles a single test at a time (of up to 5 different options). To have two tests running at a time, you'd need to upload a second script -- not especially difficult, but kind of kludgy.
The program offers two particular advantages, however. First, each of the test options is displayed using the same URL. This keeps search engines from changing the indexing of highly ranked pages and allows you to work on elements of your home page, if you like.
Second, the optional content is inserted by an SSI call on a webpage, allowing you to insert content in the midst of an existing webpage, rather than inserting the entire content of two full separate webpages.
Foley is widely known on the Internet as a careful, honest, and effective marketer. Scientific Web Marketing System is probably the first implementation of split-testing for websites. While it is now showing its age, his ConvertMoreTraffic website contains a wealth of information and tips beyond the Scientific Web Marketing System, which is essentially a bonus to the chief treasures of the site.
Sales Page Master PRO
Sales Page Master PRO
http://www.salespagemasterpro.com
Paul Hancox
Absolute Media & Marketing
37 Ivyhouse Road,
Kings Norton
Birmingham, West Midlands,
B38 8JL UK
Phone +44 121603228
This is a Perl CGI program for Unix servers that needs Server Side Includes (SSIs) enabled. It costs $89 to $149, depending upon for which offer you qualify. The program is able to track and enforce various "offer deadlines." I wasn't able to contact the developer via e-mail in order to test this product.
JavaScript Rotation Generators
Raymond McNally's Split Test Creator
Raymond McNally's Split Test Creator
http://www.splittestcreator.com
Raymond McNally
RTD Web Solutions
300 For Rd., Suite A2,
Bristol, PA 19007
Phone (215) 781-3160
$37 to $67, depending on which A/B test page you happen to see. This small .exe program installs easily on your Windows desktop computer. Here's how it works: First, create two separate webpages, each with one difference that you'd like to test,, then upload them to your server with titles such as index1.htm and index2.htm. When you click on Split_Test.exe, a screen pops up that allows you to enter the two (or more) URLs to which you'd like traffic randomly redirected. When you click a button, the program generates JavaScript code that you copy and paste into the head area of the webpage you are testing. Then upload that webpage to the server. Visitors who click on the URL will be randomly redirected to one or the other of the two URLs, and a cookie will be placed on their web browsers so if they go to the test URL again in the next 24 hours, they'll only see the webpage they've already seen. That way if you are price testing, they'll see the same price for 24 hours.
The script seems to work fine at what it does. But the program makes no record of the number of clicks to each URL or resulting sales from each. Nevertheless, if you don't have permission to install CGI or PHP programs on your server, Split Test Creator would get you started.
Ad Tracking Programs with a Split-Testing Feature
Add2it Go-To Pro
Add2it Go-To Pro
http://add2it.com/scripts/go-to-pro.shtml
Frank Bauer
Add2it.com
August-Sievers-Ring 15,
24148 Kiel, Germany
$34
This program is primarily designed to track the effectiveness of Pay Per Click advertising on Google AdWords and Overture (formerly GoTo.com, whence the name of the product). The Pro version includes an A/B split-testing feature, but it is not the focus of this product. (An ad supported free version does not include this feature.) This Perl CGI program requires a Unix or Windows server with full cgi-bin access. It requires the following Perl modules to work correctly -- CGI::Carp, Fcntl, Socket, Time::Local, IO::Socket. They note that "a good host will have them all." The web interface reminds me of Ron Wooley's free Lnkinlite program that counts click-throughs and redirects them to another URL.
ProAnalyzer
ProAnalyzer
http://www.wilsonweb.com/afd/proanalyzer.htm
Jared Deschner
Simpler Marketing
PO Box 5254, Airdrie, AB T4B 2B3, Canada
$97
This is a full ad tracking software program with a split-testing feature. The program can use either Perl or PHP for tracking -- your choice. Actual sales are recorded using a coded invisible GIF image on your thank you page so the program is capable of tracking sales from off-site e-commerce services. The program includes a single system-wide split-tester that can redirect to 2 or 3 webpages but is limited to one test at a time. This seems like a good all-around ad tracking tool that can track e-mail and other ad campaigns as well as do A/B split-testing on your website to perfect landing pages. The weakness for PPC ad campaigns is that for effective tracking you need to set up a different account to track each keyword. If you have lots of keywords this could get pretty tedious unless you track groups of keywords. Then it would work fine. The limitation of a single A/B test at a time is a drawback.
Chapters in this series:
- Introduction to A/B Split-Testing for Your Webpages
- How to Test Your Landing Pages to Improve Conversion Rates
- What to Look for in Testing Software for Small Business Websites
- A/B Split and Multivariate Testing Software Reviews (13 products reviewed)
