AB Testing

Learn more

A/B testing is a method for comparing two versions of a website, app, advertisement, message, or another digital element. Version A is typically the control, while version B contains the change being evaluated. Users are randomly assigned to one of the versions, and their behavior is compared using metrics selected before the experiment begins.

The purpose of an A/B test is not to determine which version the design team likes more. The experiment is intended to show whether a specific change affects actual user behavior, such as completing a purchase, submitting a form, creating an account, clicking a button, or using a particular feature.

What is A/B testing?

In the simplest experiment, some users see the existing version of an element, labeled A, while the remaining users receive the modified version B. Both variants run at the same time, which exposes them to similar conditions, including the day of the week, seasonality, active advertising campaigns, and temporary changes in demand.

Assignment should be random and consistent. A person will generally remain in the same group throughout the experiment. If the same user sees version A during one visit and version B during another, the results may be distorted.

An A/B test is a controlled experiment. Simply publishing two versions is not enough. Reliable testing also requires accurate data collection, an adequate sample, a clear hypothesis, and an analysis method that can assess whether an observed difference may be due to chance.

How does an A/B test work?

A standard A/B testing process includes:

identifying a problem or improvement opportunity;
reviewing quantitative and qualitative evidence;
formulating a hypothesis;
selecting a primary success metric;
preparing control A and treatment B;
defining the users eligible for the experiment;
estimating the required sample and test duration;
randomly assigning users to variants;
monitoring experiment delivery and data collection;
analyzing the result and its uncertainty;
deciding whether to launch, reject, or further develop the change;
documenting the result and lessons learned.

Both variants should be tested across relevant devices and browsers before launch. A technical defect limited to one version may influence user behavior, causing the experiment to measure the consequences of a bug rather than the intended change.

What is an example of A/B testing?

An online store wants to increase purchases and believes that more specific information on the product button will help customers make a decision. Version A uses the label “Add to cart,” while version B says “Add to cart — ships within 24 hours.”

Each variant receives 10,000 users. In version A, 500 users make a purchase, while version B generates 575 purchases. The corresponding conversion rates are 5% and 5.75%.

The absolute difference is 0.75 percentage points. In relative terms, version B performed 15% better than version A. These figures describe the same change in different ways, so reports should clearly distinguish percentage points from relative percentage improvement.

The higher result for version B does not automatically make it the winner. The team must still evaluate statistical uncertainty, data quality, effects on other metrics, and whether the improvement is meaningful to the business.

What can be A/B tested?

A/B tests can cover small interface elements as well as complete user journeys. Websites and online stores can test:

headlines and sales copy;
CTA labels, colors, and placement;
forms and the number of required fields;
product photos, videos, and graphics;
landing page layouts;
the presentation of prices, discounts, and shipping costs;
product pages and category listings;
navigation menus;
product search and filtering;
registration and checkout processes;
product availability messages;
recommendations and cross-selling;
newsletter signup forms;
localized versions of messages;
new application features.

Experiments can also be conducted in advertising, email campaigns, mobile apps, SaaS products, and onboarding flows. The underlying method is similar, but the assignment process and metrics must reflect the channel being tested.

What is an A/B testing hypothesis?

A hypothesis states what will change, which users it is expected to affect, which result it should improve, and why the effect is expected.

For example: “Displaying the estimated shipping time next to the purchase button will increase the add-to-cart rate because it reduces uncertainty about order fulfillment.”

A strong hypothesis should be based on analytics, user research, session recordings, form analysis, customer support requests, or observed friction in the customer journey. Testing random design changes may produce many results without generating useful knowledge.

If version B changes the headline, image, layout, price, and button at the same time, it becomes difficult to identify which component caused the result. Such an experiment may still be useful when comparing complete concepts, but it will not easily explain the mechanism behind the difference.

Which metrics are used in A/B testing?

Before launch, the team should define the primary metric that will be used to evaluate the hypothesis. Depending on the product, it may be:

purchase conversion rate;
revenue per user;
submitted inquiries;
account registrations;
account activation;
clicks on a specific button;
process starts or completions;
user retention;
subscription renewals;
adoption of a new feature.

Secondary metrics can provide additional context about user behavior. Teams also use guardrail metrics to detect unintended consequences, such as lower revenue, increased cancellations, more errors, slower performance, or additional customer support requests.

A variant may improve one stage of the journey while hurting the final outcome. More button clicks do not necessarily produce more purchases. The primary metric should therefore be as closely connected to the actual business objective as practical.

What is a conversion rate?

A conversion rate is the proportion of measured users who complete the desired action. It is calculated by dividing the number of users who converted by the total number of users included in the measurement.

A conversion may be a purchase, form submission, download, registration, or another action defined as a goal. However, not every experiment should rely solely on conversion rate. For an ecommerce business, revenue, margin, average order value, returns, and customer lifetime value may also be important.

What is statistical significance in A/B testing?

Statistical significance helps assess whether an observed difference would be unlikely if the tested change had no real effect. It does not automatically prove that version B is better, that the result will recur under all conditions, or that the change is profitable.

A 5% significance level is common in traditional analysis, but it is not a universal setting for every experiment. Testing platforms may use frequentist, Bayesian, or sequential methods. Results should be interpreted according to the method used by the specific platform.

In addition to a significance result, teams should evaluate effect size and a confidence interval or another measure of uncertainty. This helps determine not only whether a difference was detected, but also how large the actual effect may be.

How is statistical significance different from business significance?

With an extremely large audience, even a very small difference may become statistically significant. That difference may still generate less value than the cost of developing, launching, and maintaining the change.

Conversely, a result may suggest a valuable improvement but remain too uncertain because the sample is small. A business decision should consider:

the expected size of the effect;
the uncertainty around the result;
implementation and maintenance costs;
the effect on revenue, margin, or operating costs;
the risk of harming other metrics;
consistency with the brand strategy;
long-term consequences for users.

An experiment provides evidence for a decision, but it does not replace the entire business decision-making process.

How many users does an A/B test need?

There is no single minimum sample size that applies to every test. The required sample depends on factors such as:

the baseline conversion rate;
the smallest effect worth detecting;
the variability of the metric;
the selected significance level;
the desired statistical power;
the number of variants;
the traffic allocation between variants;
the analysis method;
the number of metrics being evaluated.

Before launch, the team can conduct a power analysis and define the minimum detectable effect, meaning the smallest improvement that would have practical value. Detecting a smaller change reliably will generally require a larger sample.

A website with limited traffic and few conversions may need to run an experiment for weeks or months. In that situation, usability testing, qualitative research, form analysis, or directly fixing an obvious problem may produce useful information more efficiently.

How long should an A/B test run?

Test duration should not be determined solely by a fixed number of days. The experiment must collect an adequate sample and cover relevant behavior cycles, which may include complete weeks, weekdays, and weekends.

A test should not be stopped immediately when the first favorable result appears. Repeatedly checking a traditional fixed-horizon test and ending it as soon as a desired threshold is reached can increase the risk of a false conclusion. Sequential testing methods operate differently, but their platform-specific rules must still be followed.

An experiment should not continue indefinitely either. Advertising campaigns, prices, competitors, devices, seasonality, and the composition of the audience may all change over time. The planned sample, minimum duration, and stopping rules should be established before launch.

Why is random assignment important?

Random assignment reduces the risk that the groups are systematically different before they encounter the tested change. If version A is shown mostly to users arriving from paid ads while version B is shown to returning direct visitors, the result may reflect traffic source rather than the variant.

The unit of assignment may be a user, account, device, session, store, company, or another unit appropriate for the experiment. The choice should reflect how the product is used. In a team-based application, assigning individual employees within the same company to different variants may cause interference between the groups.

After launch, the actual number of users in each variant should be compared with the planned allocation. An unexpected imbalance, known as a Sample Ratio Mismatch, may indicate a problem with assignment, data filtering, event tracking, or the behavior of one variant.

Does traffic always need to be split evenly?

A 50/50 allocation is common for an experiment with two variants because it uses the available traffic efficiently. However, it is not required.

A smaller percentage of users may be assigned to a new variant when the change is risky, costly, or needs a gradual rollout. Unequal allocation will generally increase the time required to reach the same level of precision. The analysis must account for the intended split.

How is an A/B test different from an A/A test?

In an A/A test, both groups receive the same product experience. Its purpose is not to select a better variant but to validate the experimentation system.

An A/A test may help identify:

random assignment defects;
differences in event tracking;
incorrect metric calculations;
an excessive false-positive rate;
integration problems between testing and analytics platforms;
data instability over time.

A lack of significant difference in one A/A test does not prove that the entire system is error-free, but it can be one part of a broader validation process.

How is A/B/n testing different from A/B testing?

An A/B/n test compares a control with several treatment variants, such as A, B, C, and D. It allows more concepts to be evaluated at the same time, but it divides traffic across more groups and increases analytical complexity.

Testing more variants also increases the chance of finding an apparently positive result by coincidence. The analysis should account for multiple comparisons or use a statistical method designed to handle them.

How is A/B testing different from multivariate testing?

A multivariate test evaluates different combinations of multiple elements. For example, it could compare two headlines, two images, and two button labels, producing eight possible combinations.

This method can estimate the effects of individual elements and interactions between them, but it will generally require substantially more traffic than a simple A/B test. It should not be confused with comparing several complete page designs, where the impact of each individual change cannot be separated.

What is split URL testing?

In a split URL test, variants are hosted at separate addresses. The experimentation system directs users from the original page to the appropriate version and then compares their behavior.

This method may be used for major layout changes, different technologies, or complete journeys spanning multiple pages. It requires careful configuration of redirects, analytics, canonical URLs, and indexing rules.

How are client-side and server-side tests different?

In a client-side test, code running in the browser modifies the completed page after it loads. This method can be relatively fast to deploy, but a poorly implemented test may cause content flicker, layout shifts, slower performance, or script conflicts.

In a server-side test, the appropriate variant is prepared before the response is sent to the user. This makes it possible to test application logic, search, recommendations, pricing, account processes, and other elements that cannot be changed safely in the browser alone.

Server-side experimentation requires more development work, but it generally provides greater control over performance, assignment stability, and the scope of the experiment.

How is A/B testing used in ecommerce?

An ecommerce experiment may cover the entire journey from a landing page to a completed order. Teams can test product presentation, filters, delivery information, the shopping cart, payment methods, forms, and cross-selling.

Purchase conversion rate should not always be the only primary measure. A change may generate more orders while reducing average order value, margin, or the number of products per order. It may also increase cancellations and returns.

Pricing and promotional experiments require particular care. In addition to statistical analysis, the business must consider transparent communication, the experience of returning customers, pricing presentation requirements, and applicable consumer protection rules.

How is A/B testing used in advertising and email marketing?

Advertising experiments can compare creative assets, copy, audiences, landing pages, and campaign settings. The groups must be comparable, and the difference should not be caused by different budgets, schedules, bidding strategies, or placements.

Email experiments may compare subject lines, sender names, copy, CTAs, layouts, and sending times. Open rates may be inaccurate because of privacy protections and automatic image loading. The evaluation should therefore also consider clicks, conversions, unsubscribes, and the actual business outcome.

How is A/B testing different from usability testing?

An A/B test primarily answers which version performs better for a defined audience and metric. It usually does not explain exactly why users behaved as they did.

A usability test involves observing people as they complete tasks. It helps uncover friction, misunderstandings, and the reasons behind user behavior. It will generally involve fewer participants but produce more detailed qualitative insights.

The two methods work well together. UX research can identify a problem and support a hypothesis, while an A/B test can measure the proposed solution’s impact across a larger audience.

How does A/B testing relate to CRO?

CRO, or conversion rate optimization, is the process of improving a website or product so that more users complete important goals. A/B testing is one of the methods used in CRO, but it is not the entire process.

Effective optimization may combine web analytics, user research, behavioral analysis, UX audits, usability testing, technical analysis, and experimentation. Regularly changing button colors without understanding user problems does not create a mature CRO program.

Can A/B testing affect SEO?

A properly implemented temporary A/B test does not have to harm a website’s search visibility. However, its technical configuration matters.

Google recommends that website experiments:

do not show Googlebot content that differs from what users receive;
use rel="canonical" for variants hosted at separate URLs;
use temporary redirects such as 302 rather than permanent 301 redirects;
run only as long as necessary;
remove testing elements and implement the selected version after completion.

An experiment should not be used to conceal content from search engines or manipulate their evaluation of a page. When testing important content, URLs, internal links, and complete templates, indexing consequences should be considered from the beginning.

How does A/B testing relate to WCAG and digital accessibility?

Every experiment variant should remain accessible to people with disabilities. Users should not knowingly be directed to a version with insufficient contrast, inaccessible keyboard controls, incorrect form labels, or screen reader barriers simply to measure the effect on conversions.

Variants should be reviewed for HTML semantics, focus order, accessible names, content resizing, responsive behavior, error messages, and the WCAG requirements adopted for the project.

An A/B test can compare different accessible solutions, but a business result does not prove WCAG conformance. If an inaccessible element increases clicks among most users, that does not mean it should be launched.

Does A/B testing use cookies and personal data?

An experimentation platform will generally need to remember which variant was assigned to a user. It may use cookies, browser storage, an account ID, a device identifier, or a server-side mechanism for this purpose.

The experiment may also connect variant assignment with analytics data, purchases, and other events. Depending on the technology, this information may constitute personal data or be linked to it.

Implementation should therefore be coordinated with the consent management platform, privacy notice, analytics configuration, and data retention rules. The organization should identify the purpose and legal basis for processing, minimize the information collected, and determine whether the provider transfers data to other countries or organizations. Specific obligations depend on the test mechanism, the data involved, and the laws applicable to the service.

What are the most common A/B testing mistakes?

Common mistakes include:

running a test without a clear hypothesis;
selecting the primary metric after seeing the result;
using too few users or conversions;
ending the test after the first favorable result;
testing too many unrelated changes at once;
comparing variants that ran during different periods;
inconsistent assignment of users;
incorrect or incomplete event tracking;
a technical defect causing an unintended sample imbalance;
evaluating many metrics without accounting for multiple comparisons;
ignoring effects across devices, channels, or user groups;
treating a nonsignificant result as proof that the variants are identical;
confusing correlation with an effect caused by the change;
launching a variant without considering its costs and side effects;
failing to document previous experiments.

Selective analysis is another common problem. If the complete sample does not show a reliable improvement, a team should not search through arbitrary segments after the test until it finds one in which version B appears to win.

When should an A/B test not be used?

A/B testing may not be the right method when:

the website has insufficient traffic or very few conversions;
the change is required for legal, security, or accessibility reasons;
the outcome cannot be measured reliably;
users cannot be assigned randomly;
the variants are likely to interfere with one another;
the cost of the experiment exceeds its potential value;
usability testing could identify the problem more efficiently;
a technical defect requires an immediate fix;
the change applies to a very small, one-time audience;
the organization does not yet have reliable analytics.

Not every decision requires an experiment. Removing an obvious barrier, repairing a broken form, or providing keyboard access should not depend on proving that the change increases conversions.

How should A/B test results be analyzed?

Analysis should begin with data quality and a comparison between the observed group sizes and the intended traffic allocation. The team can then evaluate the primary metric, effect size, uncertainty, and guardrail metrics.

It is also important to check whether:

the experiment operated as planned throughout its duration;
outages, campaign changes, or tracking defects occurred;
the variants had similar device and traffic source distributions;
the result depends on one unusual day;
the effect continues through later stages of the journey;
the change harms the experience of important user groups;
the improvement is large enough to matter to the business.

Segment analysis may provide additional insights, but it requires caution. The more data subdivisions are explored after the test, the easier it becomes to find an apparent effect by chance.

What does it mean when there is no winning variant?

A lack of a statistically reliable difference does not always mean the variants perform identically. The experiment may have insufficient power, the metric may be insensitive, or the actual effect may be smaller than the test was designed to detect.

A neutral result can still show that:

the change does not address an important problem;
users do not notice the difference;
the simpler or less expensive version can be selected;
the hypothesis needs to be revised;
additional qualitative research is needed;
a larger and more meaningful change should be tested;
the control can remain without sacrificing a detectable benefit.

A negative or neutral outcome is not a failed experiment when the test was designed correctly and provided information needed for a decision.

How can a reliable A/B testing program be created?

A single test may improve one part of a website, but the greatest value comes from a repeatable experimentation process. An organization should retain hypotheses, configurations, results, and decisions so future teams do not repeat the same experiments.

A mature A/B testing program requires:

reliable analytics and consistent metric definitions;
a process for selecting and prioritizing hypotheses;
quality assurance before experiment launch;
clearly defined decision rules;
collaboration between designers, developers, analysts, and business stakeholders;
consideration of performance, accessibility, privacy, and security;
documentation of neutral and losing tests as well as winners;
periodic verification that previous gains still persist.

A/B testing does not guarantee higher sales and cannot replace strategy, user research, or product expertise. It can, however, reduce uncertainty and provide real-world evidence about whether a specific change helps users and supports the organization’s goals.