Anatomy of Response Time and Tail Latency Amplification Diagram

Four Time Concepts

Response time: everything the client sees, end to end
Service time: how long the service actively processes
Queueing delay: waiting before processing starts
Latency: any time the request is not being processed (e.g. network)

Queueing Is the Villain

As throughput approaches capacity, queueing delay rises sharply
A few slow requests block the rest: head-of-line blocking
Measure response times on the client side, not the server side

Percentiles, Not Averages

Median (p50) shows the typical request; p95, p99, p999 show outliers
Amazon targets p99.9: the slowest users are the most valuable ones
Averaging percentiles is meaningless: add histograms instead

Tail Latency Amplification

One slow backend call makes the entire user request slow
More backend calls means a higher chance of hitting a slow one
SLOs and SLAs are written in percentiles and availability targets