A/B Testing Glossary

A controlled experiment comparing two or more versions of something (like a web page or feature) to see which performs better.

The raw difference in conversion rates (e.g., 12% vs. 10% = +2%).

The average amount of money spent per order.

The average revenue earned per visitor to your site.

An approach that estimates the probability one variant is better than another — often easier to interpret than traditional statistics.

A closer look at performance by segment to uncover patterns or outliers.

The percentage of confidence in your result — for example, 95% means you’re 95% sure the difference is real.

The original version used as a baseline to measure improvement.

When a user completes your desired action, like making a purchase or signing up.

The percentage of users who convert out of all who saw the variant.

The range that likely contains the true result based on your data.

A technique that reduces noise by accounting for pre-experiment behavior, making tests more sensitive.

A visual overview showing each variant’s performance, probability, and uplift.

When external changes (like seasonality or ad campaigns) affect your test results.

How long your test runs before results can be trusted.

A prioritized list of upcoming tests, usually based on impact vs. effort.

When a real effect is missed because of too little data or bad setup.

When a test shows a winner by chance, not because the change really worked.

A traditional statistical approach using p-values and significance levels to decide if results are real.

The metric you want to improve in your experiment — for instance, “Sign-ups per visitor.”

A metric you track to ensure the experiment doesn’t harm key KPIs (like revenue or uptime).

A control group excluded from personalization or new features to measure long-term impact.

Your prediction of what change will have a positive impact — and why.

When two changes interact and produce unexpected combined results.

How quickly your team runs and learns from experiments — a measure of experimentation culture.

Another word for improvement or gain over control.

A variant that performs worse than control.

The smallest difference you care to detect between variants.

A smarter testing method that automatically sends more traffic to better-performing variants over time.

When no variant clearly outperforms the others.

The ultimate goal your experiments support — the metric that reflects long-term success.

In Bayesian testing, this shows the full probability range of possible outcomes.

The probability your test correctly detects a real difference.

The main metric you use to decide the winner of an experiment.

The likelihood that a variant performs better than the control.

The percentage increase compared to control (e.g., 12% vs. 10% = +20%).

The total revenue divided by total sessions — a common goal metric in e-commerce.

The number of visitors or conversions you need for reliable results.

Additional metrics you track to understand side effects or extra insights.

A subgroup of your audience, such as device type, location, or traffic source.

A confidence measure showing the chance that results are not due to randomness.

A rule for when to end a test — e.g., after two weeks or when confidence exceeds 95%.

How your visitors are divided between variants, such as 50/50 or 25/25/25/25.

Another term for a variation or test version.

The percentage improvement of a variant compared to the control.

One of the versions being tested — for example, “A” (control) and “B” (new idea).

The variant that performs best based on your chosen goal or probability threshold.