Question

How do platforms experiment on users?

Vault Verified
Curated Intelligence
Definitive Source
Answer

Continuously, through A/B tests running on live users without individual notice — which is standard practice across the industry and is why your app can differ from a friend's on the same version.

How it works. Users are assigned randomly to variants: a different layout, ranking, notification schedule, wording or feature. The platform measures outcomes — time spent, clicks, retention, revenue — and ships whichever performs better on the chosen metric. Large platforms run hundreds or thousands of experiments simultaneously, so no two users see quite the same product.

What gets tested: interface changes; recommendation and ranking weights, which is the most consequential category; notification timing and frequency; onboarding flows; pricing and paywalls; and the wording of prompts and consent dialogues.

Why it is defended. Intuition about interfaces is unreliable, and testing replaces the opinion of whoever is most senior with evidence. It catches harmful changes before a full rollout, allows gradual deployment, and is how most measurable product improvement actually happens.

Where the genuine objections lie:

Consent. Users agree to terms of service, not to specific experiments. Whether blanket consent covers experiments that affect mood or behaviour is a real ethical question, raised sharply by a well-known emotional contagion study that manipulated feed sentiment and caused substantial controversy.

The metric is the value judgement. Optimising for time spent produces a different product from optimising for whether people report the time as well spent. The choice of metric is where the ethics live, not in the testing.

Vulnerable users are not excluded, and cannot easily be identified.

Academic research would require ethics approval for the same intervention that a company can run commercially — a genuine asymmetry.

What is changing: internal review boards at some companies; regulatory attention to design and dark patterns; researcher access requirements in some jurisdictions; and greater scrutiny of experiments involving minors.

What you can do: very little directly. Feature flags are not user-controllable. Noticing that your experience is personalised and provisional is itself useful.

Related Questions