Press play to start listening
Most of what gets called a scaling problem has nothing to do with traffic. My text scanning app had a stretch of about seven weeks when some scans failed with “couldn’t complete this scan,” and traffic had nothing to do with it. A model upgrade slowed scans, and the slowdown showed up everywhere the system quietly assumed the model would be fast.
This is that story, plus the general mechanics behind it. The production numbers come from my monitoring and commit history. Where I use made-up numbers to show the arithmetic, I say so.
A few terms
Scalability is whether adding resources gets you proportionally more throughput, meaning more completed work per second, without making latency worse. Reliability is whether the system still gives correct answers in reasonable time while part of it is failing.
From the outside, the two look the same. An overloaded system and a hung dependency produce the same timeout, the same error message and the same red line on a dashboard. That’s why “it doesn’t scale” gets said about failures that have nothing to do with load.
A few more terms help:
- Latency is best read as a distribution. The p50 is the typical request, and the p99 is what one user in a hundred gets.
- Saturation is work waiting in a queue. It’s different from utilization, which is how busy something is.
- Vertical scaling means a bigger machine.
- Horizontal scaling means more machines. It only helps if the bottleneck is the thing you’re adding more of.
Two formulas are worth knowing. Little’s Law says the number of requests in flight equals arrival rate times latency. And for a single server with random arrivals, mean response time is service time divided by one minus utilization. That’s twice the service time at 50% busy, five times at 80% and ten times at 90%. This curve is why a system that looks fine at 70% can fall over at 90%.
How the app works
You photograph a page. The phone resizes and encodes it, and a Firebase function sends it to a model and returns the text while you watch a spinner.
The first version used Google Cloud Vision for OCR, plus a separate OpenAI call for titles and corrections. The current one uses an LLM-based pipeline. Pages go in, and JSON comes out with the language, the raw text, a corrected version, a title, and optionally a summary and some additional insights. That’s one dependency instead of two, and far less code. On gemini-2.0-flash-lite, the daily p50 was 6.6–7.4 seconds and the p95 was 14–17 seconds, comfortably inside the function’s 60-second deadline.
A slower model
Then I moved to gemini-2.5-flash-lite, and the numbers shifted:
| All requests, daily | 2.0 Flash-Lite | 2.5 Flash-Lite |
| p50 | 6.6–7.4 s | 8–9.5 s |
| p95 | 14–17 s | 20–29 s |
| p99 | n/a | 60 s, every day for seven weeks |
The median got a little slower, but the tail is where it hurt. For seven weeks, p99 sat at the 60-second timeout every day, and in one two-week window at least 200 scans ended in a 504 at 59.99 seconds.
Retries amplified it. The retry logic suited a fast model: try a failed call twice more, and regenerate if the JSON came back malformed. That’s up to six model calls for one scan. At seven seconds a call that’s harmless. During a slow patch, though, the first attempt could take most of the minute, and each retry started with less time than the attempt before it had needed.
Two fixes
The first fix gave each request a time budget instead of a pile of independent attempts. The function’s deadline went from 60 to 300 seconds, about five times the slowest successful scan. Every model call now receives the same absolute deadline, each attempt has its own cap, and a retry only starts if at least 12 seconds are left.
The second went after the bottleneck. I ran a control benchmark, the same model on a different endpoint, and it was just as slow. That ruled out the region and pointed at the model. Moving to gemini-3.1-flash-lite roughly halved scan times. A tax document went from 13.6–15.5 seconds to 7.3–12.9, with no failures across 30 calls.
What’s different when the bottleneck is a model
The interesting lessons weren’t about timeouts. They were about what changes when the slowest thing in your system is a model you rent.
The response schema is a latency budget. Sending lower-resolution images cut input tokens by 25% and didn’t change latency at all, because generation time depends on how much the model writes, not how much it reads. Every field in the JSON is text the model produces while someone watches a spinner. The app already asks for a smaller schema when the user hasn’t requested a summary or insights, and I now treat each new field as a latency decision.
A model version is a capacity change with an expiry date. Switching versions moved p95 by several seconds without any other code changing. And you don’t fully control the timing. Hosted models get retired, and the version I’d just moved away from was scheduled for retirement less than three months later anyway. Each forced migration is effectively a load test, so it’s worth keeping a benchmark you can rerun on the next candidate.
Concurrency sets the blast radius. By default, each instance of the function serves up to 80 requests at once. One small guard in the fix handles the late failure of a model call that has already lost the race against its timer. Without it, Node 22 would treat that failure as fatal, and the process would take every in-flight scan down with it. High concurrency saves money and cold starts. It also makes the instance, rather than the request, the unit of failure.
Success-only metrics flatter you. During the slow period, the last 1,000 successful requests had a p99 of 26.9 seconds, which looks healthy. Across all requests, p99 was 60. Measuring only successes hides exactly the slow requests you care about. And when p99 equals your timeout, you’re really measuring the timeout.
Deadlines have to nest
Raising a server deadline has a side effect worth knowing about. The Android SDK’s default timeout for these calls is 70 seconds, and Flutter’s is 60. So with a 290-second server budget, a long scan can fail on the phone while the server keeps working, paying for a result nobody will see. Tap scan again and two generations run for one page. The failure also stops showing up as a server error, so it appears only in client crash reports.
The fix is to line the numbers up: the client timeout longer than the server deadline, and the server deadline longer than the total of the attempt caps. You can set the client timeout explicitly. Or you can send the client’s deadline with the request and budget everything from that, which Google’s SRE book calls deadline propagation.
The textbook part, briefly
Queueing. Little’s Law turns slowness into concurrency. With made-up numbers: at 10 scans a second and 8 seconds each, 80 scans are in flight, enough to fill one instance. If the model slows to 30 seconds, the same traffic means 300 in flight across four instances, nearly all of them waiting. Autoscaling absorbs the wait and bills you for it.
Contention. Requests that need the same row or lock take turns. If a global UPDATE stats SET scans = scans + 1 holds its lock for 5 ms, that caps you at 200 scans a second, however many servers you add (made-up numbers again). Sharded counters avoid the queue, and so does appending events and summing them later.
Retry amplification. Retries multiply across layers. Google’s SRE book describes three layers making four attempts each, which turns one user action into 64 attempts at the bottom, all arriving exactly when the dependency is struggling. To keep that in check:
- retry at one layer only
- cap retries by the time remaining
- add jitter
- keep retries under a budget (the book’s example is 10% of requests)
Connection limits catch people out on serverless. Postgres defaults to 100 connections, three of them reserved for superusers, and every instance opens its own pool. My app doesn’t use Postgres, but suppose it did, with 20 connections per instance. The slow-model scenario above pushes you to six instances and 120 connections, and every connection past the 97th fails. A pooler like PgBouncer, smaller pools and an instance cap based on the connection limit all prevent it.
When the “unscalable” choice is right
One function. I’d keep the scan path as a single function. An in-process call takes nanoseconds. A call to another service adds latency and new ways to fail, such as timeouts, dropped connections and the other service being down. Splitting the scan into services would add all of that to a request whose cost is already one model call.
Synchronous calls are fine too, as long as scans usually finish well inside the client timeout. Going async means a job store, idempotency keys and a way to push results back. It also adds new failure modes, such as a finished job whose notification never arrives. It starts to make sense for work that can outlast the client timeout, like a long multi-page scan. A queue doesn’t remove a bottleneck. It moves the waiting somewhere you control.
Relational databases stay the right choice for a long time if you need transactions, constraints or joins. Sharding gives up cross-shard joins and transactions, and before you get there you have plenty of room:
- query plans
- connection pooling
- read replicas
- partitioning by time
- a bigger instance
What I’d do next time
- Measure all requests. Include failures in latency, and keep percentiles per endpoint and timing per dependency.
- Line latency graphs up against deploy history. That’s how this slowdown was traced to a single deploy.
- Find the bottleneck and change one thing at a time against a control. Then check that the number of users feel actually moved.
- Load-test at a fixed arrival rate, for example with k6’s constant-arrival-rate. Looping virtual users slow down when your server does, which hides the problem; the effect is called coordinated omission. Then stub the model with long delays and 429s, and watch what the timeouts, retries and client do.
- Benchmark every model change as if it were a capacity change.
Most of this incident wasn’t exotic. It was a set of settings sized for a fast model meeting a slower one. The useful habit is to write those numbers down, check that they nest, and test them against the real latency distribution, failures included.
