What is A/B testing and how many visitors do I need?

An A/B test shows two versions of something to comparable groups at the same time and measures which performs better against a defined goal. Running versions sequentially instead is not a valid test, because anything else that changed in the interval, such as a seasonal shift or a marketing campaign, is indistinguishable from the effect you are measuring.

Sample size is the question people most often get wrong, and the honest answer is that it depends on two things: your current conversion rate, and how small an improvement you want to be able to detect.

The relationship is unintuitive. Detecting a large effect needs relatively few visitors. Detecting a small one needs enormously more, and the requirement grows roughly with the inverse square of the effect size, so halving the improvement you want to detect quadruples the traffic needed. A site converting at two percent, trying to detect a relative improvement of ten percent, typically needs tens of thousands of visitors per variant. Free sample size calculators do this arithmetic, and running one before starting is worth the two minutes.

The most common and damaging mistake is stopping when the result first looks significant. Statistical significance fluctuates during a test, and repeatedly checking until it crosses a threshold guarantees false positives. Decide the sample size in advance and run to completion.

Run for whole weeks, since behaviour differs by day, and a test spanning only weekdays is not representative.

The practical implication for a low-traffic site is that most tests are not worth running, because the required volume is unreachable. Testing large changes rather than small ones, or relying on qualitative research instead, is more productive than an underpowered test that produces noise.