LLM API timeout and retry handling: budgets, backoff, and what a retry re-bills

Quick answer: three budgets, one decision table.

An LLM API call is not a normal HTTP request. Generation time scales with output length, so a healthy request can legitimately run for two minutes while a stuck one looks exactly like a slow one. Generic defaults fail in both directions: a blanket 30-second client timeout kills long generations that were about to succeed, while the official OpenAI Python SDK ships with a 600-second default that can leave a dead request holding a worker for ten minutes. The fix is to split the call into phases and budget each one. Connect should finish in a few seconds from anywhere on the planet, so keep it short and unforgiving. First token, on streaming requests, proves the provider accepted and scheduled your request; give it 10 to 30 seconds depending on prompt size. The total budget is arithmetic rather than a guess: max_tokens divided by an honest decode speed, plus headroom for queueing and prompt processing. On the retry side, classify before acting. Connection failures, 408, 429, and 5xx responses are transient and deserve a second attempt with exponential backoff and jitter. Everything in the 400 to 404 range is deterministic: the same request fails the same way on every attempt, and since every attempt re-bills your full prompt as input tokens, an unclassified retry loop converts a bug directly into spend.

Where the seconds actually go.

Every completion request passes through three phases. Queueing is the wait for capacity, and it stretches when the provider is under load. Prefill is the model reading your prompt, and it scales with input length: a 100K-token context takes visibly longer to process than a 2K one. Decode is generation itself, and it is close to linear in output tokens. Time to first token covers the first two phases; everything after that is decode. Decode speed is the number that turns timeout tuning from folklore into arithmetic. Sustained speeds for hosted models commonly land in the tens of tokens per second, though the exact figure varies by model, provider load, and context length, so treat any number you have not measured as approximate. At roughly 40 tokens per second, a 4,096-token generation is about 102 seconds of pure decode. A fixed 60-second timeout does not make that request faster; it guarantees the request can never finish, and then your client re-bills the prompt by retrying it. Non-streaming requests make all of this worse, because the connection looks completely idle from the last byte of your request until the entire completion is ready. Corporate proxies and load balancers commonly cut idle connections after a minute or two, which surfaces as mysterious resets on exactly your longest, most expensive calls. Streaming keeps bytes flowing, resets idle counters along the path, and hands you a far better health signal: the gap between chunks. A stream that goes quiet for 30 seconds mid-generation is telling you something a total timeout never could. One more clock matters in production: every intermediary between you and the model enforces its own limit. If your total budget is longer than the shortest idle limit on the path, the path wins, and you see a 504 or a reset no matter what your client config says. Streaming sidesteps this too, because bytes keep moving through every hop for as long as tokens are being produced.

Worked example: turning token math into timeout budgets.

Take a RAG service that sends roughly 12K input tokens and caps output at 1,024 tokens. At a measured decode speed of about 50 tokens per second, decode alone is close to 21 seconds. Add prefill on the 12K prompt plus normal queueing, and a p95 around 30 seconds is realistic on a healthy day. A sensible budget doubles the healthy p95 rather than shaving it: connect at 3 seconds, first token at 15, total at 60. The request that needed 61 seconds was almost certainly stuck, not slow. Budget the retries into the same math. Three attempts against a 60-second total timeout means a user could wait three minutes plus backoff before seeing an error, which is usually worse than failing fast after two. Whatever latency you promise upstream has to cover attempts multiplied by the total budget, not one attempt on a lucky day. The same arithmetic produces different budgets for different workloads, which is the point: timeout values belong per route, derived from each route's max_tokens, not in one global constant.

Why retries surprise teams at the worst moment.

The common thread is amplification. A retry policy tuned on a quiet Tuesday meets its first real test during a provider incident, which is precisely when every other client on the internet is also retrying. Retry budgets, jitter, and honest error classification exist so that your service degrades gracefully instead of piling on.

• Every attempt re-bills the entire prompt. Input tokens dominate most production bills, so retries multiply the biggest line item, not the smallest.
• A client-side timeout does not reliably cancel server-side generation. The request you abandoned may finish anyway, and the provider may bill it.
• Retry layers stack multiplicatively. Two SDK retries under three queue-worker retries is up to nine attempts per user action, which is how one provider hiccup becomes a self-inflicted outage.
• Retrying a 429 without honoring Retry-After synchronizes your fleet into waves that arrive together, keeping you rate limited for longer.
• Agent and tool-calling steps are not idempotent. Replaying a step that already sent the email or wrote the row duplicates the side effect.
• Context-window overflows return 400, and no amount of retrying shrinks the prompt. Deterministic errors need code fixes, not backoff.

What to retry: the status code decision table.

Two rules cover most of it. Retry what is transient, with backoff and jitter. Never retry what is deterministic, because the outcome is already known and each attempt costs input tokens. This is the version you can paste into a runbook. One nuance on 429: the same status covers momentary rate limiting and an exhausted quota or balance. A Retry-After of a second or two means back off and go again. A quota-exhausted account keeps returning 429 no matter how patiently you wait, so repeated 429s on a fresh rate window are a billing alert, not a retry candidate.

A production retry policy in seven moves.

Failover and unit price interact, which is what the sixth move is really about. One gateway option is APIsRouter, an OpenAI-compatible API with pay-as-you-go billing and no subscription: global models are priced 20% below official list, Chinese models sit below their official rates, the first top-up adds +100% balance, and the checkout at /topup takes payment first and emails the key. The table below shows what retry amplification costs per model when a rough patch pushes your fleet to 1.2 attempts per request, one extra attempt for every five requests.

• Split the timeout into connect, first token, and total, and derive the total from max_tokens per route instead of one global value.
• Cap attempts at 2 or 3 and add a retry budget: if retries exceed a fixed share of live traffic, stop retrying and shed load instead.
• Use exponential backoff with full jitter, and always let a Retry-After header override your own schedule.
• Stream anything longer than a short paragraph. First token becomes your health check, and inter-chunk gaps expose stalls that a total timeout hides.
• Make retries safe: key each user action with a request ID, dedupe on it, and gate tool executions so a replayed step cannot repeat a side effect.
• Fail over on the second failure instead of hammering the same busy endpoint a third time, either to a sibling model or through a cheaper OpenAI-compatible endpoint where the same request costs less per attempt.
• Log every attempt with model, tokens, status, and latency, and alert on retry ratio, not just error ratio. Retry ratio moves first.

Config example: timeouts and retries that behave.

The OpenAI Python SDK already retries connection errors, 408, 429, and 5xx with exponential backoff, so the first layer is configuration rather than code: set granular httpx timeouts and a low max_retries. With stream=True, the read timeout applies to the gap between chunks rather than to the whole response, which turns it into exactly the mid-generation stall detector described above. The base URL for any OpenAI-compatible endpoint ends at /v1. If you add your own wrapper for queue workers or batch jobs, make it the only other retry layer, and keep it honest: classify status codes, respect Retry-After, and jitter everything else. Two disciplined layers with known multipliers beat five accidental ones every time. Before shipping, fault-inject. Point the client at a mock that returns 429 with a Retry-After header, a socket that hangs after connect, and a stream that dies mid-generation, then verify attempt counts and sleep durations in your logs. Retry code is exactly the code that never runs in staging and always runs during an incident.

import httpx
from openai import OpenAI

client = OpenAI(
    base_url="https://api.apisrouter.com/v1",
    api_key="sk-APIsRouter-...",
    # connect fast, tolerate 30s gaps between streamed chunks
    timeout=httpx.Timeout(connect=5.0, read=30.0, write=10.0, pool=5.0),
    max_retries=2,  # SDK backs off on connection errors, 408, 429, 5xx
)

stream = client.chat.completions.create(
    model="deepseek-v4-pro",
    messages=[{"role": "user", "content": "Summarize the incident report."}],
    max_tokens=1024,
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content or ""
    print(delta, end="", flush=True)

FAQ