What is a natural experiment?
A study exploiting a situation where circumstances have effectively assigned people to different conditions without the researcher doing it — allowing causal conclusions where a deliberate experiment would be impossible or unethical.
The problem it solves. Randomised experiments establish causation but cannot be run for most important questions. You cannot randomly assign people to poverty, to a policy regime, to a disaster, or to a school system. Observational comparison is then vulnerable to confounding, because people who differ in the exposure differ in other ways too.
A natural experiment looks for circumstances where the assignment was effectively arbitrary with respect to the outcome.
The main designs:
Difference-in-differences. Compare a group affected by a change with a comparable group that was not, before and after. Crucially, it compares changes rather than levels, so stable differences between the groups cancel out. Widely used to evaluate policies introduced in one area and not another.
Regression discontinuity. Where a threshold determines treatment — a test score cut-off, an income limit, an age boundary — people just above and just below are very similar in every respect except the treatment. Comparing them approximates randomisation locally, and this is among the most credible non-experimental designs.
Instrumental variables. Find something that affects the exposure but has no direct effect on the outcome except through it. Distance to a facility, the weather, or a lottery outcome have been used. The assumption is strong, frequently untestable, and where it fails the result is badly misleading.
Policy variation across borders, and staggered introduction of a change.
Well-known examples: studies comparing minimum wage changes across neighbouring US states; comparisons of twins separated by circumstance; and evaluations of health policies introduced in one nation of the UK before others.
The assumptions that must hold: for difference-in-differences, that the two groups would have followed parallel trends without the change — which cannot be proven, only supported by the pre-period; and for regression discontinuity, that nothing else changes at the threshold and that people cannot precisely manipulate which side they fall on.
The strength of a natural experiment rests entirely on those assumptions, which is why they are argued about more than the results.