A/B Testing

A/B testing randomly splits a user pool into two groups, shows each a different variant and uses statistical inference to decide which version actually wins. The discipline is in 'statistically significant', not 'looks better'.

Common ways A/B tests fail in practice include calling them too early on a small sample, watching too many metrics so random variance gets read as real lift, and letting self-selection break the groups. Writing the experiment design down before launch prevents most of these.

In low-traffic environments — most B2B contexts qualify — A/B tests often can't reach significance in any reasonable timeframe. Qualitative interviews, Bayesian methods or simply moving faster on judgement calls usually beat over-engineered split tests there.