Case Study: Conversion Rate Optimization using Offermatica

by Dr. Ralph F. Wilson, E-Commerce Consultant
Web Marketing Today Premium, Issue 82, Summer 2004 Summary: A/B split-testing and Taguchi Method experiments designed to optimize the conversion (subscription) rate on forms to subscribe to a free Internet marketing newsletter. As a result of testing, the subscription rate in one form was increased 37%, another 25%. Offermatica.com was the testing system used.

---

This article contains older information. Go here for newer information on conversion and testing.

Executive Summary: Using Offermatica.com for A/B split-testing and Taguchi Method experiments, I was able to raise the conversion rate in my newsletter subscription forms from 25% to 37%, enabling me to turn around a subscriber erosion of 23% over 8 months.

One of the biggest issues facing me was to stop the hemorrhaging of newsletter subscriptions from my Web Marketing Today Free Edition. Over a 8-month period, from January to August 2004, my list of active subscribers dropped 22.6% -- from 112,400 to 87,000. The main problem wasn't people unsubscribing because they didn't like the content. I've always had some minor attrition this way, but in the past new subscriptions had always more than made up for losses. The main problem was three-fold: (1) people changing e-mail addresses to get rid of overwhelming spam, (2) bounces from spam filters that couldn't distinguish my double-opt-in newsletter from spam, and (3) the increased use of pop-up blockers. In the first three quarters of 2003, 75% of my new subscribers came through a pop-up window that appeared 5 seconds after they came to any page of my website. But by the first three quarters of 2004, less than half of my new subscribers came through pop-up windows. The pop-up window produced an approximate 3% subscription (conversion) rate, compared to a 1.5% conversion rate for subscriptions from the form at the bottom of the page.

In September it turned around. I began to slowly gain subscribers again as a result of two main factors: (1) some help from a co-registration program and (2) substantial help from optimizing my on-site subscription forms that resulted in raising my conversion rates 25% to 36%.

Co-registrations make up about 13% of my current new subscribers. Not counting co-registration subscriptions, subscriptions from my website itself are divided into subscribers from my pop-up window (43%) and from a form at the bottom of each page (57%). I decided to work at optimizing both of these subscription forms.

Choice of Offermatica

After studying all the split-testing tools available to small to medium businesses, I had selected two best-of-breed tools: Vertster, which redirects visitors to the test webpages, and Offermatica, which displays the changed content into the actual webpages being tested. (You can read reviews of various testing software options in my e-book How to Optimize Your Landing Pages Scientifically.) Vertster's redirect approach works fine to optimize landing pages that aren't designed for search engine traffic. But I needed to test elements that affect nearly every page of my website -- without confusing search engine rankings of my webpages with redirects. I needed the test content inserted into these webpages, which is easily done with Offermatica's mBox system that retrieves the changing content from Offermatica's server using JavaScript calls on my webpages.

Subscription Form at Bottom of Page

The subscription form at the bottom of nearly every page in my site is now the chief source of subscribers, but it was converting only about 1.5% of visitors who saw it. If I could raise the conversion rate, this would make a big difference. From early studies on the pop-up subscription form it seemed that the presence of a graphic showing the three free e-books that came with a subscription tended to depress subscriptions. So I set up a three-way split-test. 

1. Subscription form (with e-books graphic) below the sponsored links at the bottom of the page (default).
2. Subscription form (without e-books graphic) above the sponsored links.
3. Subscription form (with e-books graphic) above the sponsored links.

I ran the test from June 28 to July 10, 2004 and found that the best result was Configuration C. Apparently, the sponsored links between the end of an article and the subscription form had the effect of distracting readers from subscribing. The graphic of the e-books helped the subscription rate (unlike my experience on the pop-up window described below). On July 10 I ended the test and changed the configuration of my webpages to Configuration C, with a 37.4% increase in conversion (subscription) rate over the previous configuration -- a huge result determined with a rather simple two-week test.

	Visitors	Conversions	Conversion Rate	Increase over default	% Confidence Level
A. Below sponsored links, includes 3 free e-books picture. (default)	11,082	172	1.55%	default	99%
B. Above sponsored links, no picture	10,028	181	1.80%	16.1%	99%
C. Above sponsored links, includes 3 free e-book picture.	10,113	215	2.13%	37.4%	99%

Yellow Highlighting

However, not all tests yield such a spectacular result. With this same subscription form in Configuration A (described above), I experimented with and without yellow highlighter around the words "includes 3 e-books" -- expecting that the yellow highlighter would attract the eye more effectively to a key benefit of subscribing, increasing the subscription rate. I was wrong. From  June 22 to June 28, 2004, based on 15,000 visitors, no statistically significant difference in conversion rate was observed. To save clutter, I dropped the yellow highlighting.

	Visitors	Conversions	Conversion Rate	Increase over default
Yellow highlight	7,626	154	2.02%	Not significant
No highlight (default)	7,594	153	2.01%	default

 Taguchi Analysis of Pop-Up upon Entering Site WMT Subscription Box

At the same time as I was optimizing the subscription form at the bottom of the page, I was studying how to optimize the conversion (subscription) rate with the yellow-background pop-up window that had served me well for several years -- and used to account for 75% of my subscribers. Though pop-up blockers are decreasing the number of visitors who see this form, it still accounts for a significant number of subscriptions and boasts a significantly higher conversion rate than the bottom-of-page form.

Two staff members from Offermatica -- technical engineer Mark Cyster and vice president Michael Carrier -- developed a series of multivariate experiments using the Taguchi Method. This method was used to reduce the number of experiments needed to optimize the form. Four areas of the form were selected for testing with and without various features:

1. A drop-down menu requiring the subscriber's country
2. A graphic showing free e-books available to subscribers
3. A photo of me (Ralph Wilson)
4. The words "If you decide you want to get off this list later, one click will unsubscribe you. Don't worry."

All but the last element were included in the yellow-background pop-up window I had been using for more than a year. 

You can view both the original unoptimized pop-up subscription form and the final optimized pop-up subscription form. They don't look that much different, but the optimized window gets a 25% higher conversion (subscription) rate. I had expected the graphics to raise the conversion rate, thought the "easy unsubscribe" line to help significantly, and asking for country information to slightly depress conversion rates. Since the results seemed counterintuitive, the test was run twice with the same results. The first time from June 15-24, 2004 with 8,550 visitors and the second for 18 days, from June 25-July 15, 2004 with 16,812 visitors. Analysis for the second test is shown below: 

The chart below shows which factors were most important contributing the conversion rate either positively or negatively. Factor D, "easy unsubscribe" text, didn't have enough data to be statistically significant. When the pop-up window subscription form was redesigned according to the results of these tests, a 25% improvement was predicted.

Factor Included	Helped Conversion	Hindered Conversion	Contribution of Factor	% Confidence Level
Include country dropdown menu	Significantly		35.0%	88%
Include free books picture		Significant	28.8%	86%
Include picture of Dr. Wilson		Mild	12.0%	71%
"Easy unsubscribe" text	Very mild		1.8%	34%

The graph below shows the percent contribution to the conversion rate of the various factors tested.

Optimized Pop-Up Boxes Upon Entering and Upon Leaving

After optimizing the pop-up window with the yellow background, I experimented with an entirely new design -- a white background with a picture of several young marketers at the top. Then I compared these as windows that pop up 5 seconds after first arriving at the site vs. windows that pop up upon leaving the site. One of these pop-up-upon-leaving windows included a short sound clip appealing to the visitor to subscribe. Here's what I found with this test that ran from July 30 to Sept 27, 2004: (Note: this doesn't add up to total site visitors, since repeat visitors have a cookie that keeps them from seeing the pop-up again.)

The graph below shows percent conversion rates for the various options tested.

Event		Visitors	Conversions		Conversion Rate	Confidence Level
Pop-Up 5 seconds after Arrival
	A. Optimized Yellow Bkg (default)	17562		661	3.76%	Not statistically different
	B. New White Bkg	17560		643	3.66%	Not statistically different
Pop-Up upon Leaving Site
	A. New White Bkg	10125		303	2.99%	95%
	B. New White Bkg w. sound	10164		340	3.35%	95%
	C. Optimized Yellow Bkg (default)	10201		356	3.49%	95%

The conclusions show an overall slightly better result for an initial pop-up over a pop-up upon leaving. However, the difference between optimized pop-up (yellow background) as an initial pop-up vs. a pop-up upon leaving is not statistically significant with the amount of current data.

Sound seemed to increase the conversion rate significantly in the pop-up upon leaving scenario. It still remains to be seen if sound would help the optimized yellow background pop-up significantly upon leaving. 

From these experiments I've learned that when visitors see something can make a significant difference in how they respond to it. The same information when they arrive is more effective than when they leave. I've also learned that what works best is not always intuitive -- even to people who have been doing web marketing for a long time. 

Your situation will be different. But I hope that this case study will help you see how to approach your problem. By all means test!

---

Other articles from Web Marketing Today Premium, Issue 82, Summer 2004