What is regression to the mean?
The tendency for an extreme measurement to be followed by a less extreme one, simply because extreme values partly reflect chance — and failing to account for it produces confident conclusions about interventions that did nothing.
Why it happens. Any measurement combines a stable underlying value and random variation. An unusually high or low reading is likely to have had an unusually favourable or unfavourable contribution from the random part. On the next measurement, the random part is unlikely to be as extreme again, so the value moves back toward the average — regardless of anything done in between.
The originating observation. Francis Galton noticed that tall parents tended to have children shorter than themselves, and short parents children taller — "regression towards mediocrity". The children were not being pulled toward average by any force; the extremes of one generation partly reflected chance combinations.
Why it constantly produces false conclusions:
Selecting on an extreme value guarantees the appearance of improvement. Take the worst-performing schools, the sickest patients, the most accident-prone drivers or the worst-performing branches, apply any intervention, and measure again. The group will improve on average. A substantial part of that improvement would have occurred with no intervention at all, and without a control group there is no way to separate the two.
The reward and punishment trap, described by Kahneman. If you praise exceptional performance, the next attempt is usually worse; if you criticise poor performance, the next attempt is usually better. The natural inference — that criticism works and praise does not — is exactly backwards, and it is drawn from regression alone.
Sports commentary about slumps and hot streaks, and the "curse" attached to appearing on a magazine cover, which follows an exceptional season.
Medical improvement after treatment sought at the point of worst symptoms, which is also why many conditions appear to respond to whatever was tried.
What protects against it: a control group selected the same way; randomisation; and measuring change against what would be expected without intervention rather than against the starting point.