High-throughput APIs: how to stay responsive when traffic spikes
Practical guide to APIs that stay fast under load: p95 and p99 latency, caching, paging, queues, rate limiting, idempotence and load testing, with concrete examples.

An API is fine as long as ten people use it. The problems come when a campaign, a new integration, or a partner sends requests in a burst, and the response time goes from 80 milliseconds to four seconds. The good news: most bottlenecks repeat themselves from project to project and are solved with a set of known practices. In the article I go through what you measure, where time is wasted and what you can do, in the order in which I would do it.
In short
- Measures percentiles (p95, p99), not mean. Users remember their slowest request.
- Most gains come from the database: fewer queries, matching indexes, cached results.
- What not to do while the client is waiting is moved to a queue and processed in the background.
- Limits (rate limiting, timeouts, maximum size) protect the system from its own clients.
- Without load tests you are left with guesswork. A test done once a quarter is worth more than any estimate.
It starts with measurement, not optimization
The first question is not “how do I make it faster?”, but “how fast is it, for whom and when?”. Average response times lie. If 95 out of 100 requests take 100ms and 5 take 5 seconds, the average shows 345ms, which is “okay”, but one in twenty users waited too long.
That’s why you watch percentiles:
- p50 (median): typical experience.
- p95: what one in twenty users feel.
- p99: the long tail, where the most unpleasant surprises and usually the most interesting causes occur.
In addition to latency, you track error rate, number of requests per second, and resource saturation (CPU, memory, database connections). Together they tell you if the system still has room to breathe or is close to the limit.
An explicit goal helps: “p95 under 300ms for read endpoints, under 800ms for write endpoints, 99.9% successful requests”. Without it, any discussion of performance becomes opinion.
The database is usually the culprit
When an API slows down, in almost all projects I see the same order of causes: the database, then calls to other services, then the code itself. Some classic problems:
Problem N+1. You bring in a list of 50 orders, then for each order you do a separate query for the customer. You have 51 queries where two would have been enough. It is solved by loading the relations at once (join or grouped query).
Indexes missing. A query on a non-indexed field traverses the entire table. It runs fine with 1,000 rows and crashes at 5 million. Most databases have an execution plan analysis tool (eg EXPLAIN) that shows you exactly where the entire table is read.
Fields fetched for nothing. SELECT * on a table with large columns (long texts, JSON) transfers data that the response doesn’t even use. You only ask for what you send.
Connections. Each request that opens a new connection pays a cost of several tens of milliseconds, and the database supports a limited number of simultaneous connections. A head-sized connection pool solves both problems.
For frequent reads with data that rarely changes (catalogs, settings, price lists), the biggest gains come from cache.
Cache: what, where and for how long
The cache is the most powerful and most dangerous tool. Powerful, because a request served from memory costs microseconds. Dangerous, because old data served with trust produces bugs that are hard to detect.
A practical hierarchy, from client to server:
- Cache in browser and CDN, controlled by
Cache-ControlandETagheaders. Public and identical answers for all (product lists, articles) can be served without ever reaching your server. - Application cache (eg Redis): Expensive query results, kept seconds or minutes.
- Database cache and materialized views for heavy reports.
The rule that saves you trouble: you decide from the beginninghow to invalidate. You have two simple strategies. First, timeout (TTL): “this list can be no more than 60 seconds old”. Second, invalidate on write: when a product changes, you delete its entry from the cache. For many cases, a combination of them is sufficient.
An example header for a public resource that can sit in the cache for 60 seconds, with 5 minutes of tolerance while refreshing in the background:
Cache-Control: public, max-age=60, stale-while-revalidate=300
ETag: "a1b2c3"
Beware of custom data: User-dependent responses are not publicly cached, otherwise one client may receive another’s data. For them you use private or exclude them completely.
Pagination and size-controlled responses
An endpoint that returns “all orders” works in test and fails in production when there are 200,000. Any list needs pagination, and choice matters:
| Method | How it works | When you choose it |
|---|---|---|
Offset (?page=3) |
Skip N rows | Small lists, page number interfaces |
Cursor (?after=id) |
Continues from the last item seen | Large lists, continuous scrolling, changing data |
Offset paging slows down as you go deeper, because the database counts and discards skipped rows. Cursor pagination remains constant and does not produce duplicates when new items appear. For public or high-volume APIs, I recommend it by default.
Adds a maximum page size (eg 100) that the client cannot exceed, even if requested. The same principle applies to the body size of incoming requests.
What should not be done synchronously, is done in queue
Many slow endpoints do too many things before they respond: send emails, generate PDFs, recalculate statistics, call three external services. The client is waiting all this time, and a single slow external service is holding all your requests.
The healthy model: The API validates the request, saves it, queues a task (Redis, RabbitMQ, SQS, etc.) and immediately responds with 202 Accepted and an identifier. A separate process (worker) executes the task. The client learns the result via a status endpoint or a webhook.
The advantages are multiple: fast responses, tasks that can be resumed if they fail, capacity that scales independently (add workers without touching the API), and a buffer that absorbs traffic peaks. If 10,000 requests come in a minute, the queue holds them and the workers process them at their own pace.
Boundaries: Protect the system from its own clients
A client with the wrong loop can bring down an entire API with no ill intent. Limits are your safety net:
- Rate limiting. A maximum number of requests per minute for each API key or IP address. On overflow you respond with
429 Too Many Requestsand theRetry-Afterheader, which tells the client when to try again. - Timeouts everywhere. Every call to database or external service has a time limit. Without it, a dead dependency blocks the entire chain.
- Circuit breaker. If an external service fails repeatedly, it is not called for a while and the API responds with a fallback or clear error instead of crawling.
- Controlled degradation. When the system is under pressure, you first stop the secondary functions (recommendations, statistics) and keep the core (orders, payments).
Idempotence: retries without duplicates
In a distributed system, requests are lost and clients retry. If a payment was sent, the response got lost along the way, and the customer resubmits the request, you risk charging twice. The solution is idempotency: the same request, sent several times, produces only one effect.
Basically, for operations that modify data, the client sends a unique key (Idempotency-Key). The server stores it along with the result. On a retry with the same key, it returns the saved result instead of performing the operation again. It’s a small detail to implement and saves you from some of the most costly incidents.
Scaling: First better, then more
The reflex is to add servers. It works, but it costs and moves the problem. The order I follow:
- Eliminate unnecessary work: extra queries, fetched fields for nothing, repeated serializations.
- Add cache where data allows.
- Move to background what not to do during request.
- Scale horizontally service, with multiple stateless instances behind a load balancer. Sessions are held outside the instance (Redis, tokens), otherwise a new instance cannot take over the user.
- Scale out the database: read replicas for heavy queries, partitioning or domain separation only when really needed.
Each step costs more than the previous one in complexity. The first three solve, in my experience, the vast majority of cases encountered in SMEs and growing products.
Test the upload before users do
A load test simulates realistic traffic and shows where the system is failing. Tools like k6, Locust or Gatling do this without much effort. A useful scenario has four phases: warm-up, gradual increase, plateau at the expected level, and a two- to three-fold peak to see how it decays.
What you watch during the test: at what number of requests per second p95 exceeds the goal, which resource saturates first, and how the system behaves when exceeded (gradually slows down or crashes suddenly). The answer to the last question is the most valuable. A system that gradually slows down and gives you time to react is much better than one that crashes without warning.
Run the test in an environment as close to production as possible and repeat it after big changes. The old results do not apply to the new architecture.
Conclusion
A fast API under pressure doesn’t require exotic technologies, it requires discipline: you measure percentiles, clean up queries, cache with an invalidation strategy, move the hard work to the background, set limits, do idempotent operations, and load test. Applied in order, these practices take an API from “runs while nobody uses it” to “remains predictable when it succeeds.”