What is a circuit breaker, and how do systems fail gracefully?
A circuit breaker stops calling a failing dependency for a while, so a slow or broken service does not take down everything that depends on it. It exists because the default behaviour — retrying immediately and repeatedly — makes outages substantially worse.
The failure it prevents. Service A calls service B. B slows down. A's requests pile up waiting, consuming threads or connections. A becomes unresponsive, so C, which calls A, also fails. One slow dependency cascades into a full outage, and the retries from every layer add load precisely when B can least handle it.
How a circuit breaker works. It tracks failures and has three states:
Closed — normal, calls pass through, failures counted.
Open — after a threshold, calls fail immediately without being attempted, returning an error or fallback in microseconds. This sheds load from the struggling service and keeps the caller responsive.
Half-open — after a cooldown, a small number of trial requests are allowed. Success closes the circuit; failure reopens it.
The companion patterns, which matter as much:
Timeouts on everything. An unbounded wait is the root cause of most cascades. A call with no timeout is a bug, and the default in many clients is no timeout at all.
Bulkheads, partitioning resources so one dependency cannot consume the whole connection pool or thread budget.
Retries with exponential backoff and jitter. Fixed-interval retries from many clients synchronise into a thundering herd; jitter spreads them out.
Retry budgets, capping retries as a fraction of traffic.
Load shedding, rejecting excess work early rather than degrading for everyone.
Graceful degradation, returning stale cached data, a reduced feature set or a sensible default instead of an error — the difference between a recommendations outage and a homepage outage.
Idempotency, without which retries are unsafe.
The principle underneath: decide in advance what to do when a dependency is unavailable, because it will be. Failing fast and partially beats failing slowly and completely.