System Design · Architecture under constraints
What is the difference between latency and throughput?
Short answer
Latency is how long one operation takes, usually measured at a percentile. Throughput is how many operations the system completes per unit of time. Improving one can worsen the other.
Latency is a duration: a request takes 40 milliseconds at the median and 250 milliseconds at p99. Throughput is a rate: the service handles 12,000 requests per second. A system can have low latency and low throughput, or high throughput with long queues and terrible tail latency.
Little's Law connects them: the average number of requests in the system equals arrival rate times average time in the system. If 1,000 requests per second arrive and each spends 200 milliseconds in the system, about 200 requests are in flight on average. That number is what sizing and queue depth calculations are built on.
Why average latency misleads
- Averages hide the slow requests that users actually remember
- Queueing delay grows quickly as utilization approaches capacity
- One slow dependency can dominate p99 even when the median is fine
- Batch and background work can raise throughput while increasing per-request latency
Batching is the classic trade-off. Combining writes into larger batches improves throughput and lowers per-record cost, but each record waits for the batch window. Caching lowers read latency and increases effective throughput until the cache misses, at which point the origin sees a burst.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How does Little's Law help with capacity planning?
- 2Why does p99 latency rise before CPU is fully utilized?
- 3When is batching the right trade-off?
- 4How can adding replicas reduce latency but not throughput?
Say it in the interview
Latency is the time a single request takes, and throughput is the rate of completed work. I care about tail latency because the average hides the slow requests users notice. Little's Law lets me connect arrival rate, in-flight requests and time in system, which is useful for sizing queues. Batching, caching and replication all trade one of these metrics against the other.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live System Design mock interview and get scored feedback on depth, clarity and confidence.