A/B testing
Comparing two or more variants against live traffic split at random, so the difference in results can be attributed to the change rather than to circumstance. The honest core of performance culture - and honest only with enough volume, a single variable, and the discipline not to peek early.
Where it's used
Used on sites, emails, ads and products. Good for answering a specific question with real behaviour, when you have enough traffic.
How to calculate it
Conversion rate = Conversions ÷ Visitors
Relative lift = (Variant rate - Control rate) ÷ Control rate
Worked example
Illustrative numbers10,000 visitors are randomly shown each checkout button. The current button gets 300 orders, and the new one gets 345.
Control
300 orders ÷ 10,000 visitors = 3%
Variant
345 orders ÷ 10,000 visitors = 3.45%
Relative lift
(3.45 variant - 3 control) ÷ 3 control = 15%
What it tells you
Run a significance test before shipping. A gap like this is near the edge at this sample size, and stopping as soon as it looks good is how false wins get made.
