System Design · Architecture under constraints
How would you design rate limiting across a distributed service?
Short answer
Choose a limit dimension and algorithm, enforce it with an atomic shared counter such as Redis, and return 429 with retry metadata. Handle hot keys, clock skew and a deliberate fail-open or fail-closed policy.
Rate limiting protects a service from overload and enforces fair use or billing tiers. The first decision is the dimension: per user, per API key, per IP, per tenant or per endpoint. The second is the window: requests per second, per minute or a burst allowance on top of a sustained rate.
Algorithms
- Fixed window: simple counters reset on a boundary, but allows a burst twice the limit across the boundary
- Sliding window log: precise but stores every timestamp, which is expensive at high volume
- Sliding window counter: interpolates between two windows, a good accuracy and memory compromise
- Token bucket: refills at a steady rate and allows bursts up to the bucket size, common for APIs
- Leaky bucket: smooths output at a fixed rate, useful for shaping rather than admission
In a distributed deployment, local counters are not enough because each instance would allow the full limit. Use a shared store with an atomic operation, such as a Redis script that reads, refills and decrements in one step. Avoid read-modify-write across two round trips, which races under concurrency.
Hot keys are the main scaling problem. A single tenant with enormous traffic can overload the Redis shard holding its counter. Mitigate with local pre-checks, sharded counters that sum to the limit, or a dedicated limiter for large tenants. Decide explicitly whether the limiter fails open when Redis is unavailable. For abuse protection, fail closed; for availability-first APIs, fail open with an alert.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1Why does a fixed window allow a burst at the boundary?
- 2How do you make the counter update atomic?
- 3Should the limiter fail open or closed?
- 4How do you handle a single tenant with a hot counter?
Say it in the interview
I start with the limit dimension and the burst policy, then choose an algorithm such as token bucket for API traffic. Enforcement has to be atomic in a shared store, because per-instance counters would multiply the limit by the fleet size. I plan for hot keys and clock skew, and I decide explicitly whether to fail open or closed during a limiter outage. The response includes 429, Retry-After and metrics so clients and operators can react.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live System Design mock interview and get scored feedback on depth, clarity and confidence.