So, what is A/B testing and how can you start implementing it on your own website and online store? In this article, we’ll cover everything you need to know about A/B testing and share testing ideas to help you get started.
What is A/B testing?
At its most basic, A/B testing, also known as split testing, is a way to compare different versions of something to see which performs better. In these experiments, you define a conversion goal to measure, like clicks or completed transactions. Two variations of the same marketing asset (like a web page or email) are then shown to different users at random while measuring the difference in performance.
For example, let’s say you wanted to increase the number of clicks on the “Buy now” button on your product pages. You could run an A/B test to find out how button color affects click-through rates, experimenting with a green button and a blue button. You would:
Define your conversion goal. In this example, you want to measure click-through rate.
Design the A/B test. How big of a sample size do you want? Who will participate, new customers or existing customers?
Gather data. Will you run your own test or use A/B testing software? For how long will the test run?
Analyze the results. Which variation had the biggest positive impact on the conversion metric that matters most?
At the end of the A/B test, you’ll be able to confidently implement the winning variation without worrying about jeopardizing conversions.
Pro tip: Use a landing page builder to create pages that target different audiences throughout your funnel. Then A/B test them to optimize them for success.
Why you should A/B test your online store
A/B testing is often used across traditional marketing strategies and campaigns to increase email open rates or boost social media engagement. However, when you apply these same principles to your online store, the benefits are even more impactful. Simply generating a 1% increase in clicks on your buy button could result in thousands of dollars in extra sales.
Eliminate guesswork
Designing your online store can be an extremely subjective process, with everyone having their own visual preferences or making suggestions to mimic industry leaders. By continuously running different A/B tests, you can offer a data-driven approach to design and remove subjectivity from the process.
Customize design decisions for your business
Homepage design, product pages, and the checkout experience look different for every business depending on the items sold and target audience. And, what works for one store won’t necessarily work for yours. A/B testing allows you to generate results that are hyper-targeted and specific to your business and customers.
Experiment in a low-risk environment
Changing even a few design elements on your homepage can create a jarring experience for returning customers. A/B testing allows you to make these types of changes in a controlled environment without sacrificing long-term customer loyalty, retention or brand awareness.
A/B testing examples
The most challenging part of A/B testing is often coming up with the experiments themselves. Which pages should you focus on? Which elements should you change? How big of a change should you make?
Sometimes, you’ll be able to react to customer feedback. For example, if customers keep asking the same questions about your return policy, you could test different ways to highlight relevant return policy details (like including a link to your return policy vs. including the full policy details directly on the product page). Other times, you have to get creative and come up with experiments yourself.
01. Storefront testing ideas
Seasonal items vs. evergreen products
Should you curate relevant, timely items on your storefront or let customers find what they’re looking for based on their own needs and interests?
Example: Istanbul Cats, an organization aimed at “popularizing the love of animals,” chooses to feature timely items, showcasing their back-to-school products during the summer.
Multiple product images vs. a single hero image
Do your customers prefer to see a variety of products right on the homepage or is it overwhelming? Example: Jérome Studio, a Berlin-based leather bag company, takes a minimalist approach and features a stylized hero image of only one of their items.
02. Category page testing ideas
Best-sellers vs promotional items
When customers land on a category page, are they interested in the most popular products or items on sale? Example: Simplu, a Portuguese organic food company, chooses to highlight items that are 15% off.
Carousel vs. static images
How do your customers like to browse: do they want to scroll down a page and see all the items or do they want a curated set of items that rotate through a carousel? Example: Onno, a terrarium and plant store, allows customers to browse all their items at once on each category page.
03. Product page
Long-form vs. short-form copy
How much information do customers want to digest on the product page? Are they interested in the story behind the product or tips on how to care for the item? Example: Handlebend adds extra content to their product detail pages, including how their products are made, their size and weight, and a fun description that encourages customers to buy.
Buy button color
How does your “Buy now” or “Add to cart” button color affect click-through rates? Should you use a bright color to catch customers’ attention or try to match the color palette of the rest of your store? Example: Studio007, a photography studio, opts for an attention-grabbing red color.
04. Checkout experience
One-page vs. multi-step
Do customers prefer to see all the steps in the checkout process in one place, so they know what to expect? Or, do they like to focus on one step at a time, progressing to a new page for each phase? Example: The Tea Story shows an overview of the entire checkout experience on one page, moving customers through different parts as they fill in their information.
Payment method options
How do your customers prefer to pay? With a credit or debit card or using another form of payment? Example: Box and Bow, a gift box shop, lets customers pay with PayPal in addition to cards.
A/B testing software
Robust, self-service A/B testing software means you don’t need to work with developers to run experiments nor rely on your business intelligence team to analyze the results. Instead, marketers, designers, and product managers can set up new tests in minutes and easily interpret results.
The one caveat, however, is that no-code A/B testing software can be a bit limiting depending on what you want to test. For example, all A/B testing tools allow you to change visual elements on your pages (like images, copy, buttons, etc). However, if you want to run tests within your checkout flow (like experimenting with upselling), you may need to work with developers to add additional code. But, for the majority of tests on your online store, third-party software will do the job.
Some of the most popular A/B testing tools include:
Visual Web Optimizer (VWO): VWO allows you to create and run tests without a single line of code. Using a visual editor, you can change headlines, CTA, or images, modify existing pages, or make layout changes. You can also use VWO’s enterprise-grade server-side testing product to conduct more complex tests deep within your store’s workflows and across devices.
Crazy Egg: Test your website, landing pages, product pages, and more with Crazy Egg. Simply add a unique Crazy Egg snippet to a page, choose the elements to test, and monitor results in real time. Crazy Egg also offers complementary features like heat maps and individual session recordings so you can better understand your visitors’ behavior.
Optimizely: Use Optimizely’s industry-leading visual editor to run multiple tests on the same page, deliver different variations to visitors based on ad campaign, geography, cookies, or other factors, and segment experiments by device or browser. You can also run A/B tests on all your channels, including mobile apps and messaging platforms, with Optimizely Full Stack, a set of open-source SDKs for product teams.
Google Optimize: Google Optimize combines the power of Google Analytics with A/B testing to help you run personalized website tests for free. Using Google Optimize’s WYSIWYG editor, you can test multiple elements on the same page, experiment with split URL testing, deploy your own tests with server-side experiments, and immediately launch the winning version of your site.
Omniconvert: To run more advanced A/B tests, Omniconvert offers a developer-focused tool to run tests with HTML, CSS or Javascript. Experiment with real-time dynamic variables like cart total value, visitor name, customer lifetime value, customer location, and more. You can track as many goals as you want on one page by calling a single line of code.
UsabilityHub: While UsabilityHub, a remote user research platform, doesn’t offer traditional A/B testing features, it does include a variety of unique ways to test different design elements on your website. You can test the effectiveness of links and content hierarchy by measuring how many people complete certain tasks, ask users to compare different design variations, measure users’ first impressions, and run design surveys. You can invite your own customers to participate in tests or recruit from UsabilityHub’s participant panel of more than 170,000 users.
How to determine an A/B test winner
The value of these A/B testing tools lies in their simplicity and ease of use — at the end of your experiment, the winner is clearly announced on the criteria you established. And with most testing tools, that’s where the analysis ends. A winner is determined by the performance of one factor, without always considering the entire funnel.
But it’s important to avoid blindly implementing results from a standalone test. That’s because while they can offer valuable directional data points (like uncovering trends or patterns), they often don’t paint the full picture.
Let’s say you define an A/B test winner based on homepage click-through rate and version A wins. Upon closer analysis, however, you notice that while version A has more clicks, it actually results in more abandoned carts long term. In this example, implementing the winning version from an A/B test would do more harm than good.
To determine the true winning variation, look at complementary data. Make sure to understand how your A/B test affected:
Different customer segments, such as new vs. returning customers, visitors from mobile vs. desktop, or high spenders vs. low spenders
Other high-value actions on your store, such as button clicks, link clicks, visits to a specific page, form submissions, or page engagement
Sales, including the number and types or products sold and average order value
In addition, look for ways to get more qualitative data to understand your A/B test results. Analyze heat maps to see exactly how visitors navigate your site and where they click or get confused. And, if possible, consider running a survey or user test to get more in-depth, anecdotal feedback.
A/B testing for long-term success
A/B testing should not be a one-and-done activity. While there is value in running one-off A/B tests based on specific needs (like testing a time-bound promotion or changing tests based on the season), this experimental mindset should be incorporated into your long-term strategy. Ideally, you would always be running a series of tests across your online store, both to try new things and reconfirm existing hypotheses. (After all, a winning test variation from 12 months ago may no longer be valid!)
The bottom line: Trying to identify the “perfect” design for your online store can be overwhelming and filled with subjectivity. With A/B testing, you can get one step closer to designing the most optimal site for your business and customers. So when in doubt, run a test.
Ready to start selling online? Create your free eCommerce website today.