Last updated: March 2025
Quick Answer
An A/B test is statistically significant when there is a 95% or higher probability that the difference between variants is real and not due to chance. Always wait for significance before making decisions.
Key Takeaways
- ✓ Statistical significance at 95% confidence means only a 5% chance the result is due to luck
- ✓ The lift shows the percentage improvement of variant B over the control
- ✓ Results below 95% confidence should not be acted on — keep running the test
- ✓ Larger sample sizes produce more reliable results
What Is A/B Testing?
A/B testing (also called split testing) compares two versions of a webpage, email, or ad to determine which performs better. You split your audience randomly between version A (control) and version B (variation), then measure which achieves a higher conversion rate.
How Statistical Significance Works
Just because variant B has a higher conversion rate doesn't mean it's actually better. Statistical significance tells you the probability that the observed difference is genuine, not random variation. The industry standard is 95% confidence.
How to Run an Effective A/B Test
1. Define Your Goal
Pick a single primary metric: conversion rate, click-through rate, or revenue per visitor.
2. Calculate Sample Size
Most tests require at least 1,000 visitors per variation. The smaller the expected improvement, the more visitors you need.
3. Run the Test Long Enough
Run for at least one full business cycle (1–2 weeks) to account for traffic patterns. Don't peek and stop early.
4. Test One Variable
Change one element per experiment for clear, actionable insights.
What Can You A/B Test?
- Headlines and copy — Often the highest-impact changes
- Call-to-action buttons — Text, color, size, and placement
- Landing page layout — Long-form vs. short-form, video vs. image
- Pricing and offers — Test price points, discounts, and free trial lengths
Frequently Asked Questions
What is statistical significance?
Statistical significance tells you the probability that the difference between two variations is real and not due to random chance. A confidence level of 95% means there is only a 5% chance the result is due to luck.
How many visitors do I need for an A/B test?
It depends on your baseline conversion rate and the minimum detectable effect. Generally, you need at least 1,000 visitors per variation, but high-traffic sites can get results faster.