How to fix AI API 500, 502, and 524 errors

Quick answer: three different failures, three different fixes

A 500 is the API telling you something broke server-side, either inside the gateway or somewhere upstream at the model provider. A 502 is more specific: the gateway itself answered, but whatever it forwarded your request to sent back a response it could not parse, or nothing at all. A 524 is different again. Nothing necessarily broke; the request just took longer to answer than the connection was willing to wait, which is common with long prompts, large context windows, or big outputs requested all at once without streaming. Treating all three the same wastes time. Retrying a 524 without shrinking the request just times out again. Retrying a 500 without checking whether it repeats on a different model burns attempts chasing a route that will not recover on its own. What actually works is layered: retry with backoff for transient failures, fall back to a second model or endpoint when a route stays unhealthy, and log enough detail on every failure that debugging does not start from memory.

Where each error actually comes from

The first three look identical from the client: a failed HTTP call. But only 500 and 502 are genuinely retryable in the sense that the same request might succeed on the next attempt. A 524 usually will not, because whatever made it slow (a long prompt, a large requested output, no streaming) is still true on attempt two. Fixing a 524 means changing the request, not repeating it. 429, 401, and 403 are worth separating out early because they look like server errors on the surface but are not. Confusing a quota problem with a broken upstream, or a key problem with a flaky provider, sends debugging effort in the wrong direction.

• 500 (internal error): the gateway hit an unexpected exception, or the upstream model provider returned a failure that could not be translated into a clean response.
• 502 (bad gateway): the gateway reached an upstream service but got back something invalid, empty, or malformed, often during a provider-side incident or an overloaded route.
• 524 (timeout): the connection closed before a response arrived, almost always because the request or the expected output was too large for a synchronous call.
• 429 (rate limited): a different failure mode entirely, capacity or quota rather than a broken response; retrying immediately just adds to the queue.
• 401/403 (auth or permission): the request never reached model inference at all, so no amount of retrying helps until the key or model access is fixed.

Worked example: a backoff schedule that will not make things worse

Retrying too aggressively is its own failure mode. A burst of retries during a real upstream incident just adds load to a route that is already struggling. Exponential backoff with jitter spaces attempts out so a brief blip clears on its own, and a real outage stops absorbing extra traffic instead of getting hammered by it. The table below runs the math for four attempts using wait = 2^attempt seconds plus up to one second of random jitter, a common and safe default.

Why these errors catch teams off guard

Most of this stays invisible until traffic is real. A handful of manual tests during development rarely hits a slow provider hour, a rate limit, and an expired key at the same time, so the code path that handles failure tends to get written last, if at all, and it shows the first time production traffic actually exercises it.

• Retrying looks like it worked. A 500 clears on attempt two, so the underlying fragility, no logging, no fallback, one hardcoded model, never gets fixed until it fails during a demo.
• Non-streaming calls hide how close a request is to timing out. A call that takes fifty-five seconds against a sixty-second budget looks fine in testing and fails the first time a user pastes in a longer document.
• 429 gets treated like 500. Backing off harder does not fix a quota problem; the fix is a slower client or a higher limit, not more retries.
• Auth errors get mistaken for outages. A 401 that started right after a key rotation looks identical to a provider incident until someone actually reads the error body.
• One model having a bad hour looks like the whole API is down, when a fallback to a second model would have kept the app answering.
• Nobody logs enough to tell these apart later. Without the model name, endpoint, and status code saved at failure time, every incident starts from zero.

500 vs 502 vs 524 vs 429 vs 401/403, side by side

Same symptom on the client, a failed request, but five different underlying causes and five different correct responses.

Practical fixes to put in front of the model call

None of this replaces reading the actual error body. But a team that logs consistently, retries selectively, and keeps one fallback model configured turns most of these incidents into a five-minute fix instead of a fire drill.

• Log every failure with the model name, endpoint, timestamp, HTTP status, and whether streaming was on. This one habit turns "it breaks sometimes" into a pattern you can actually act on.
• Retry 500 and 502 with exponential backoff and jitter, capped at three or four attempts. Do not retry 401/403, and do not retry a request that already failed for being malformed; fix the request instead.
• For 524s, shrink the request before retrying it: cap max output tokens, trim what gets sent as context, and turn on streaming for anything that can generate more than a short reply.
• Keep a short fallback list of two or three models for any call on a critical path, so one unhealthy route does not take the whole feature down with it.
• Handle 429 on its own path, separate from error handling. A rate limit needs a slower client, not a faster retry loop.
• If you want to swap models or route around a bad provider without redeploying, put an OpenAI-compatible gateway such as APIsRouter behind the same base_url pattern; changing which model or endpoint a request hits becomes a config change instead of a code change.

Retry, fallback, and a base URL you can point anywhere

The pattern below wraps a chat completion call in the backoff schedule from the table above, then falls through a short list of models if the first keeps failing. Point base_url at any OpenAI-compatible endpoint, such as https://api.apisrouter.com/v1, and swap in catalog model IDs such as deepseek-v4-flash, glm-5, or claude-sonnet-4-6 for the fallback list. The second snippet does the same thing across a list of endpoints, for the rarer case where a whole route is unhealthy rather than one model.

import random
import time
from openai import OpenAI

client = OpenAI(
    api_key="sk-APIsRouter-...",
    base_url="https://api.apisrouter.com/v1",
)

def call_with_backoff(messages, model="claude-sonnet-4-6", max_retries=4):
    for attempt in range(max_retries):
        try:
            return client.chat.completions.create(
                model=model,
                messages=messages,
                max_tokens=800,
                timeout=60,
            )
        except Exception as exc:
            status = getattr(exc, "status_code", None)
            if status not in (500, 502) or attempt == max_retries - 1:
                raise
            wait = (2 ** attempt) + random.random()
            time.sleep(wait)

FAQ