Retry Storms: How Good Clients Take Down Healthy Servers

A retry is the most reasonable line of code you will ever write. The request failed, the network is flaky, so you try again. It works on your machine, it works in the demo, it ships.

Then one day a downstream service has a bad thirty seconds, and your retry takes the whole system down with it. Not the failing dependency, your retry. The clients were healthy. The server they hammered was on its way back to healthy. The retries made sure it never got there.

This is the caller-side half of resilience. Rate limiting protects a service from the traffic it receives; retries are about the traffic your clients generate, and under stress a naive retry policy generates exactly the wrong amount at exactly the wrong time. This walks from the one-line retry to the handful of patterns that make retries safe: backoff, jitter, budgets, and circuit breakers. Go for the examples, but the reasoning is the point.

The naive retry, and why it is a trap

Here is the retry everyone writes first. Try, and if it fails, try again a few times.

func Call(ctx context.Context, do func() error) error {
    var err error
    for attempt := 0; attempt < 3; attempt++ {
        if err = do(); err == nil {
            return nil
        }
        time.Sleep(100 * time.Millisecond) // wait a bit, then try again
    }
    return err
}

On a single client hitting a mostly-healthy server, this is fine. The trap only shows up at scale, and it shows up all at once.

Picture a service with 10,000 clients. A dependency behind it hiccups for two seconds: a GC pause, a failover, a brief lock. Requests start failing. Now every one of those 10,000 clients does the same thing at the same time: it fails, waits 100ms, and retries. You have just turned one wave of traffic into four waves (the original plus three retries), and the three extra waves land in a tight cluster right when the server is least able to absorb them.

The server was about to recover. Instead it gets hit with 4× its normal load, falls further behind, fails more requests, and triggers more retries. That is a retry storm: a feedback loop where failure creates load and load creates more failure. The dependency's two-second blip becomes a twenty-minute outage, and nothing in your logs points at the real culprit, because the culprit is your own retry policy.

Two things are wrong here, and they need two different fixes:

1. The retries come too fast and too many, piling load onto a struggling server.
2. The retries come in lockstep, because every client failed at roughly the same moment and waits the same fixed delay.

Fix one: exponential backoff

The first fix is to stop retrying at a fixed, aggressive interval. Wait longer after each failure, so a server that is struggling gets progressively more room to breathe.

func Call(ctx context.Context, do func() error) error {
    const base = 100 * time.Millisecond
    var err error
    for attempt := 0; attempt < 5; attempt++ {
        if err = do(); err == nil {
            return nil
        }
        // 100ms, 200ms, 400ms, 800ms, ...
        backoff := base * (1 << attempt)
        select {
        case <-time.After(backoff):
        case <-ctx.Done():
            return ctx.Err()
        }
    }
    return err
}

Doubling the wait after each attempt (base × 2^attempt) means a client that keeps failing backs off fast: 100ms, 200ms, 400ms, 800ms. The total retry pressure on the server drops sharply compared to a fixed 100ms hammer, and a genuinely down dependency is not pestered every 100ms forever.

This is a real improvement. It is also not enough, and the reason is subtle: backoff fixes how often one client retries, but it does nothing about all clients retrying together. Ten thousand clients that failed at the same instant will still retry at the same instant, at 100ms, then again at 200ms, then 400ms. The waves are further apart now, but they are still waves. The server sees a thundering herd, just a more spaced-out one.

Fix two: jitter (this is the important one)

The missing ingredient is randomness. If every client computes the same backoff, backoff alone just reschedules the stampede. Add a random component and the herd smears out across the interval instead of arriving as a spike.

The version that works best in practice is full jitter: don't wait the backoff, wait a random duration between zero and the backoff.

func Call(ctx context.Context, do func() error) error {
    const base = 100 * time.Millisecond
    var err error
    for attempt := 0; attempt < 5; attempt++ {
        if err = do(); err == nil {
            return nil
        }
        // full jitter: sleep a random point in [0, base*2^attempt)
        window := base * (1 << attempt)
        wait := time.Duration(rand.Int63n(int64(window)))
        select {
        case <-time.After(wait):
        case <-ctx.Done():
            return ctx.Err()
        }
    }
    return err
}

It looks like a small change, one rand call, but it is the difference between a spike and a smooth trickle. With ten thousand clients each sleeping a random point in [0, window), their retries spread evenly across the window instead of landing on the same millisecond. The server sees a gentle, continuous stream it can actually serve.

It is worth being precise about why full jitter beats the tempting middle ground of "backoff plus a little noise" (window/2 + rand(0, window/2), often called equal jitter). Equal jitter still guarantees every client waits at least half the backoff, so half the interval is empty and all the retries bunch into the second half. Full jitter uses the whole interval. AWS's well-known analysis of this found full jitter both completed work faster and hammered the server with far fewer competing calls. When in doubt, full jitter.

Backoff plus full jitter fixes the timing problem: retries are now spaced out and desynchronized. But there is a deeper problem that no amount of clever timing solves.

The problem timing cannot fix: retries multiply load

Step back and count. Every retry is, by definition, extra load. A policy that allows up to 3 retries can, in the worst case, triple the traffic a service receives. And the worst case is not rare, it is exactly the moment a dependency is failing, because that is when retries fire. Retries add the most load precisely when the system has the least capacity to serve it.

This is the insight that separates people who have run large systems from people who have not: under overload, retrying makes things worse, not better. A single client retrying a transient blip is helping itself at negligible cost. Ten thousand clients retrying a systemic outage are collectively mounting a denial-of-service attack on a service that is already down. Backoff and jitter shape when the load arrives; they do nothing to cap how much.

So you need a mechanism that looks at the aggregate and says: we are retrying too much, stop.

Fix three: retry budgets

A retry budget caps retries as a fraction of live traffic. The rule, popularized by Google's SRE practice, is simple: allow retries only up to something like 10% of your successful request rate. When failures are rare, 10% is plenty and every failed request gets its retry. When a dependency is melting down and everything is failing, the budget is exhausted almost immediately, so the vast majority of failures are not retried. Retries are throttled hardest exactly when they would do the most damage.

// A retry budget: retries may not exceed `ratio` of recent successful calls.
type Budget struct {
    mu        sync.Mutex
    ratio     float64 // e.g. 0.1 → retries capped at 10% of successes
    successes float64 // decaying count of recent successes
    retries   float64 // decaying count of recent retries
}

func (b *Budget) allowRetry() bool {
    b.mu.Lock()
    defer b.mu.Unlock()
    if b.retries >= b.successes*b.ratio {
        return false // budget exhausted, do not retry, fail fast
    }
    b.retries++
    return true
}

func (b *Budget) onSuccess() {
    b.mu.Lock()
    b.successes++
    b.mu.Unlock()
}

(In production the counts decay over a rolling window rather than growing forever, but the shape is this.) The point is that the budget turns "should I retry?" from a per-request decision into a fleet-aware one. One client can no longer contribute to a storm, because the storm exhausts the shared budget and further retries are simply refused. A failed request that loses the budget lottery fails fast, which is almost always better than a retry that piles onto an outage.

Fix four: circuit breakers

Backoff, jitter, and budgets all still attempt the call. But if a dependency has been down for thirty seconds, every attempt is a wasted request: a connection, a timeout, latency your caller pays, and one more packet thrown at a service that needs fewer, not more. A circuit breaker stops attempting entirely once failures cross a threshold.

It is a three-state machine:

• Closed: normal. Requests flow through. Count failures.
• Open: too many failures. Reject immediately without calling the dependency at all. This is the key move: you fail fast locally and give the dependency total silence to recover.
• Half-open: after a cooldown, let a single trial request through. If it succeeds, close the circuit and resume. If it fails, open again and wait longer.

type Breaker struct {
    mu          sync.Mutex
    failures    int
    threshold   int           // consecutive failures before opening
    state       string        // "closed" | "open" | "half-open"
    openedAt    time.Time
    cooldown    time.Duration
}

func (b *Breaker) Call(do func() error) error {
    b.mu.Lock()
    if b.state == "open" {
        if time.Since(b.openedAt) < b.cooldown {
            b.mu.Unlock()
            return errors.New("circuit open: failing fast")
        }
        b.state = "half-open" // cooldown elapsed, allow one trial
    }
    b.mu.Unlock()

    err := do()

    b.mu.Lock()
    defer b.mu.Unlock()
    if err != nil {
        b.failures++
        if b.state == "half-open" || b.failures >= b.threshold {
            b.state = "open"
            b.openedAt = time.Now()
        }
        return err
    }
    b.failures = 0
    b.state = "closed"
    return nil
}

The breaker's real job is not to protect the caller from latency, though it does that. It is to protect the dependency from a caller that would otherwise keep knocking on a locked door. An open circuit is the caller-side equivalent of the server dropping load: both sides agree to send less traffic so the system can heal.

And the cheapest fix of all: retry less

Every technique so far assumes the request is worth retrying. Often it is not, and the biggest wins come from not retrying in the first place.

• Only retry idempotent operations. Retrying a GET is safe. Retrying a POST /charge can double-bill a customer. If the operation is not idempotent, either make it idempotent (idempotency keys) or do not retry it.
• Only retry retryable failures. A 503 or a connection timeout might succeed next time. A 400, a 404, or a 401 will fail identically every time; retrying them is pure waste that only feeds the storm.
• Bound the total time, not just the attempt count. Honor the caller's deadline. A retry that fires after the user has already given up helps no one and still costs the server a request.

Half of retry safety is just refusing to retry things that should never be retried.

The layers, side by side

None of these is sufficient alone. Backoff without jitter reschedules the stampede. Jitter without a budget smooths an unbounded amount of load. A budget without a breaker still throws wasted calls at a dead service. They compose into a policy, and a mature client uses all of them.

Takeaways

• A retry is load that arrives when the system is weakest. That single reframe explains every pattern here: the whole game is controlling how much extra load you generate during a failure, and when.
• Backoff controls how often one client retries; jitter controls whether all clients retry together. You need both, and full jitter beats every half-measure.
• Under overload, retrying is an attack on your own infrastructure. A retry budget bounds the blast radius by capping retries as a fraction of live traffic, throttling hardest exactly when it matters.
• A circuit breaker lets the caller fail fast and gives a dead dependency silence to recover. Failing fast is a feature, not a fallback.
• The cheapest retry is the one you never send. Retry only idempotent operations, only on retryable errors, and only inside the deadline.

Rate limiting and retries are the two sides of the same coin. The server sheds load it cannot serve; the client sends less load when the server is hurting. Get both right and a bad thirty seconds stays a bad thirty seconds, instead of becoming an outage that everyone remembers.