What is A/B Testing, and Why Should I Care?
If you’re like most small and mid-market business I’ve worked with, you know a little bit about A/B testing and conversion rate optimization (CRO). It’s conceptually pretty simple, with different visitors seeing different variations of your site. Each test is an experiment: which experience is more effective? Which variant leads to more on-site conversions – the leads, product views, adds to a shopping cart, checkouts, and other actions that drive your business? A well designed program can provide the answers, and improve your e-commerce performance.
If that’s the case, why haven’t you started a testing program already? You may know that testing site changes is best practice in e-commerce, but perhaps you believe that it’s only for large companies. Perhaps you and your team believe that testing changes is difficult and expensive, or that the benefits are not worth the investment. I’m here to tell you: that’s not the case. Any company can benefit from well-designed experiments. It’s relatively easy and inexpensive to get started, and the benefits may surprise you.
This guide will show you how to get started with A/B testing in the SuiteCommerce Advanced environment. We’ll also give you a few examples of tests and results that have made a difference.
Choosing a Platform
Many companies offer testing software: Optimizely, Convert, Google Optimize, Dynamic Yield, and many others. All of these are strong options, but if you’re just now running your first test (and if you use Google Analytics), I recommend starting with Google Optimize. While most of these tools run anywhere from $500 to $3k or more per month, Optimize offers a free version. Recently, Google increased the limits on the free offering to five simultaneous tests (up from three), which will be sufficient for at least the early stages of a new test program.
Setting Up Optimize with SuiteCommerce Advanced
Tavano includes setup as part of our standard A/B Testing & CRO offering. However, if you want to set up Optimize yourself Google has a detailed guide. SuiteCommerce Advanced (SCA) does have its own idiosyncrasies, however, so I have a few additional tips for you.
Just like SCA requires your Google Analytics code to be implemented in Tag Manager (GTM), the same is true for Optimize. You may read protestations to the contrary in Google’s setup instructions, and while using Optimize – ignore them, and just set it up inside GTM. There’s even a Google Optimize tag type inside GTM to help.
Since SCA is a single-page app, it’s very different from using Optimize on most other platforms.
The default option for triggering experiments is on page load:
This won’t work in SuiteCommerce Advanced, since there is only one actual page load for the entire session. Even worse, many views in the app rewrite their content based on SCA’s functionality and events – for example, when lazy-loading additional products, or when adding or deleting items from the shopping cart, or when loading content from Site Management Tools. All of those rewrites can erase your experimental changes.
The solution is to activate on a custom event instead:
It’s usually possible to create all of the necessary custom events within GTM. In some cases, however, your developer may need to make code changes to your site to ensure that Optimize’s experimental changes are applied consistently after SCA changes the page content.
Examples of Experiments, or: “What Should I Test?”
The short answer: any aspect of the user experience that’s worth changing is also worth testing. Even minor changes to the micro-copy on the site can have substantial impact.
For example, examine these variations on the product detail page of client Toad&Co:
As you can see, the only difference among the variants is the size selector micro-copy: Select vs. Select Size vs. Select your Size. Amazingly, this one detail among the variants caused substantial improvements. Within Optimize, views show both the range of expected improvement across all the goals defined for your test:
In addition, Optimize shows the details and statistical significance for each goal. In this case, one variant improved conversion rate by over 25% and revenue per session by over 35%!
In my next post, I’ll provide more examples of real-life tests and results, including improvements to the home page, product listing pages, and product detail pages. The third post in this series will cover in-depth analysis of test results in Google Analytics, including variation by audience segment and secondary metrics such as average order value.
About the Author
Jeff Rawlings has run digital marketing and conversion optimization programs for over two dozen apparel, footwear, and lifestyle brands – including his own, Koolaburra, which he sold to Deckers Outdoor Corporation in 2015. Most recently, he worked with Tavano to improve Toad&Co’s e-commerce site with extensive A/B testing following their relaunch on the SuiteCommerce Advanced platform. Jeff began his e-commerce career in 1995 as webmaster and internet marketing manager for America Online’s small business software and hosting services NaviSoft, NaviService, and PrimeHost. He writes about business, sustainability, and e-commerce on Medium and Twitter.
