How do retries and backoff work without making an outage worse?

A retry helps when a failure is temporary: a connection reset, a timeout or a 503. It hurts when the failure is permanent or when every client retries at once. The goal is to recover from transient faults without creating a retry storm.

async function withRetry(fn, attempts = 4) {
  for (let attempt = 0; attempt < attempts; attempt += 1) {
    try {
      return await fn();
    } catch (error) {
      if (!isRetryable(error) || attempt === attempts - 1) throw error;
      const base = 100 * 2 ** attempt;
      const delay = base + Math.random() * base;
      await new Promise((resolve) => setTimeout(resolve, delay));
    }
  }
}

Rules

• Retry timeouts, connection resets, 429 and 502 or 503, not 400 or 422
• Use exponential backoff with full or decorrelated jitter
• Respect Retry-After when the server provides it
• Cap total attempts and total elapsed time
• Make the operation idempotent or pass an idempotency key
• Limit retries as a fraction of traffic so an outage does not triple load

Retries stack across layers. A client retries the gateway, the gateway retries the service, and the service retries the database, multiplying the original request. Decide where retries belong, usually one layer close to the failing dependency, and make the others fail fast.