LLM fallback architecture: how to fail over without failing users

Quick answer: classify first, then retry, reroute, degrade.

Fallback is not one mechanism, it is a ladder. The first rung is a bounded retry against the same model, which absorbs transient blips like a stray 500 or a dropped connection. The second rung is rerouting to a different provider, key, or region, which is what actually saves you during a rate-limit wall or a provider outage. The third rung is degrading to a cheaper or faster model, so users get a slightly worse answer instead of an error page. The last rung is a non-LLM answer: a cached response, a canned message, or a job pushed onto a queue for later. None of the rungs work unless the error classifier in front of them is right. Retrying a 400 burns latency on a request that can never succeed. Retrying a 429 against the same exhausted key deepens the throttling. Failing over on a content-policy refusal replays the same refusal across three more providers at full token price. Classification decides which rung a failure lands on, and it is the part most homegrown fallback logic gets wrong. The rest of this guide works the design end to end: the failure classes and what each deserves, worked cost and availability math for a realistic three-tier chain, the mistakes that surprise teams in production, a comparison of strategies, and a Python implementation you can adapt.

The failure classes and what each one deserves.

Two implementation details matter as much as the classes themselves. First, separate transport errors from API errors. A connection reset before any bytes arrive is safe to retry, but a client-side timeout on a request the provider already accepted may have completed and billed anyway, so blind retries on long generations can pay for the same answer twice. Second, make the classifier data rather than scattered if-statements. A single table mapping status code plus provider error string to an action (retry, reroute, degrade, fail fast) is unit-testable, auditable, and becomes the shared vocabulary of your incident reviews. When the chain misbehaves, the first question is always "what did the classifier decide", and a table answers it in one lookup.

• 429 rate limit: back off and retry once on the same provider if the limit is per key, reroute immediately if the whole provider is throttling. A Retry-After header, when present, tells you which case you are in.
• 500, 502, 503, and overloaded errors: retry once with jitter, then reroute. This is the classic transient class, but these errors cluster during incidents, so cap attempts hard.
• Timeouts and stream stalls: treat the provider as degraded and reroute. A model that takes a minute to produce its first token is down for interactive purposes, whatever the status page says.
• 400 invalid request, 401 and 403 auth failures, 404 model not found: never retry, never reroute blind. These are your bugs or your config, and no amount of failover fixes a wrong model ID.
• Context-length errors: not transient. Either truncate and retry once with a smaller prompt, or route deliberately to a model with a larger window. Generic fallback just repeats the overflow.
• Content-policy refusals: a completed request, not a failure. Rerouting refusals through the chain multiplies cost and latency for an answer that will keep refusing.

Worked example: what a three-tier chain costs and buys.

Assume a production assistant averaging 2,000 input and 400 output tokens per request. The chain: claude-sonnet-4-6 as the primary, gpt-5.4 as the cross-vendor fallback, and deepseek-v4-pro as the degraded tier for budget or latency trips. Suppose that in a normal month 97 requests in 100 are served by the primary, 2 by the fallback, and 1 by the degraded tier. The blended cost lands at about $9.50 per 1,000 requests versus $9.60 for a primary-only setup. The fallback tiers are approximately free at the token level: the fallback model is priced close to the primary, and the degraded tier is far cheaper. What the chain actually buys is availability. If each tier independently serves 99.5% of the requests offered to it, a two-tier chain only fails when both tiers fail, which is roughly 1 request in 40,000 instead of 1 in 200. Adding the third tier pushes user-visible failure into rounding-error territory. The real cost of fallback is engineering time: the classifier, the budgets, the monitoring, and the per-tier eval pass. The token bill barely notices.

Why fallback systems surprise their owners.

The common thread: fallback is a distributed-systems problem wearing an AI costume. The mechanisms that keep service meshes alive, circuit breakers, latency budgets, jittered backoff, health probes, are exactly the ones that keep an LLM chain healthy. Teams that treat it as three lines of try/except end up designing the real system live, during their first serious provider incident.

• Retry storms: when a provider throttles, every client retries at once, and naive retry loops turn a partial brownout into a full outage. Jitter and attempt caps are not optional extras.
• Slow failures beat fast failures: a hard 503 fails over in milliseconds, but a hung connection eats the entire latency budget before the chain even starts. Aggressive connect and first-token timeouts matter more than retry counts.
• Streaming moves the decision point: you cannot swap providers halfway through a stream a user is already reading. Failover triggers on connection failure, first-token timeout, or a stalled stream, and recovery means restarting the generation on the next tier.
• Same prompt, different model: tool-call formats, JSON reliability, and instruction-following differ across families, so a fallback that works in the happy path can still break downstream parsers. Every tier needs its own eval pass before an incident, not during one.
• Silent cost shifts: one misclassified error or mis-set flag can quietly route most traffic to a fallback tier for weeks. Alert on fallback rate, not just error rate, or the bill finds out before you do.
• Double billing on timeouts: a request that times out client-side may still complete and bill server-side. On long generations, blind retries buy the same output twice.

Fallback strategies compared.

There is no single correct strategy, only layers with different protections and costs. Most production systems combine the first four rows and reserve hedging for the small slice of traffic with strict latency objectives.

Seven habits of fallback chains that survive incidents.

On the fifth point, a single endpoint that carries multiple model families turns chain design into a config exercise. APIsRouter is one such gateway: an OpenAI-compatible API with Claude, GPT, Gemini, DeepSeek, GLM, and Kimi models behind one base URL, pay-as-you-go with no subscription, global models priced 20% below official list and Chinese models below their official rates. The first top-up adds +100% balance, and the checkout at /topup takes payment first and emails the key, with no signup form. Whether you use a gateway or three separate SDKs, the architecture above stays the same; the gateway simply deletes the per-provider plumbing and unifies billing for every tier.

• Write the error classifier as a table and unit-test it. Every status code and provider error string maps to exactly one action: retry, reroute, degrade, or fail fast.
• Give each request one total latency budget and make every attempt spend from it. Two retries with 20-second timeouts is a 60-second worst case that no interactive user will sit through.
• Cap same-provider retries at two, with exponential backoff plus jitter. Beyond two attempts you are amplifying the incident, not surviving it.
• Put a circuit breaker in front of each provider: trip after a burst of failures, probe periodically, and stop offering doomed requests to an endpoint that is down.
• Standardize every tier on one OpenAI-compatible API shape, so failover is a base URL and model-string change instead of an SDK migration. Routing the chain through a cheaper OpenAI-compatible endpoint also lowers the blended per-token rate while you are at it.
• Log provider, model, attempt count, and failover reason on every request, and alert on fallback rate. A chain quietly running on tier two for a week is a working system hiding a broken one.
• Break it on purpose: revoke a staging key, black-hole a provider endpoint, inject 429s and 503s. The first execution of your fallback path should not happen during a production incident.

Implementation: a fallback chain in under 60 lines.

The core is small: an ordered list of tiers, an error classifier, a per-request deadline, and jittered backoff. The example uses the OpenAI SDK against an OpenAI-compatible endpoint, so each tier is just a model string and cross-vendor failover needs no extra clients. Two production notes. Grow the classifier by mapping provider error bodies, not only status codes, and keep that mapping in one tested module. And wrap each tier in a circuit breaker so a provider that has failed continuously for the last minute is skipped without burning an attempt from the latency budget. Before wiring the chain into an application, smoke-test every tier with the same key so a config problem cannot masquerade as an outage:

import random
import time

from openai import OpenAI, APIConnectionError, APIStatusError, APITimeoutError

client = OpenAI(
    base_url="https://api.apisrouter.com/v1",
    api_key="sk-...",
    timeout=20.0,     # per-attempt cap; tune to your latency budget
    max_retries=0,    # this module owns retries, disable the SDK's
)

CHAIN = ["claude-sonnet-4-6", "gpt-5.4", "deepseek-v4-pro"]
RETRYABLE_STATUS = {408, 409, 429, 500, 502, 503, 504}


def classify(err: Exception) -> str:
    if isinstance(err, (APITimeoutError, APIConnectionError)):
        return "reroute"  # slow or unreachable: go to the next tier
    if isinstance(err, APIStatusError):
        if err.status_code in RETRYABLE_STATUS:
            return "retry_then_reroute"
        return "fail"     # 4xx config errors: retrying cannot succeed
    return "fail"


def complete(messages: list[dict], deadline_s: float = 45.0):
    start = time.monotonic()
    for model in CHAIN:
        for attempt in range(2):          # max 2 attempts per tier
            if time.monotonic() - start > deadline_s:
                raise TimeoutError("request latency budget exhausted")
            try:
                return client.chat.completions.create(
                    model=model,
                    messages=messages,
                    max_tokens=800,
                )
            except Exception as err:
                action = classify(err)
                if action == "fail":
                    raise
                if action == "reroute":
                    break                 # skip remaining attempts on this tier
                time.sleep(min(2 ** attempt + random.random(), 8))
    raise RuntimeError("all fallback tiers exhausted")

FAQ