AI API gateway vs direct integration: how to choose

Quick answer: direct for one provider, gateway for two or more.

Direct integration means your service talks to each model provider's own API: OpenAI's endpoint with an OpenAI key, Anthropic's endpoint with an Anthropic key, and so on. You own the SDK versions, the auth, the retry policy, the rate-limit handling, and the billing relationship for each one. A gateway inverts that. Your service talks to a single OpenAI-compatible endpoint with a single key, and the gateway routes each request to the right provider based on the model ID in the payload. For a single-provider product the direct path is genuinely simpler. One SDK, one status page to watch, no extra hop in the critical path, and immediate access to provider-specific beta features the compatible surface has not absorbed yet. The calculus flips at the second model family. Every additional provider you integrate directly is a full new integration surface, not just a new key, and the operational load grows linearly while the product value of each addition shrinks. That is why most production teams converge on gateway-shaped architecture even when they never adopt a product called a gateway: they build an internal routing layer that normalizes providers behind one interface. The real decision is rarely gateway versus no gateway. It is whether you build that layer, self-host an open-source one, or use a hosted endpoint that already exists.

What each architecture costs you in practice.

Start with what direct integration actually obligates you to own per provider. Auth and key rotation against that provider's console. An SDK or HTTP client pinned and upgraded on that provider's release cadence. Retry and backoff logic tuned to that provider's rate-limit semantics, which differ in meaningful ways: status codes, retry-after headers, and overload behavior are not standardized across vendors. Streaming quirks, since server-sent events are implemented slightly differently everywhere. An error taxonomy mapping, so your application can treat "model overloaded" and "context too long" consistently no matter who raised them. Usage logging and cost attribution, because each provider's billing export has its own schema. And an on-call runbook entry for that provider's outages. None of this is exotic engineering, but it multiplies. Two providers roughly doubles the surface. Four providers means four different 429 behaviors in your retry code and four billing CSVs in your finance pipeline. Swapping a model for a cheaper one stops being a config change and becomes a small integration project, which in practice means it does not happen and you keep overpaying by default. A gateway concentrates all of that in one place. Model choice becomes data, a string in the request, instead of code. Retries, failover, per-key budgets, and spend metering are implemented once and apply to every provider behind the endpoint. The trade is real too: you add one network hop to the critical path, you take a dependency on the gateway's availability, and with a hosted gateway your traffic transits infrastructure you do not operate. Those costs are measurable and worth weighing honestly, which is what the comparison table below is for.

Worked example: one month of production traffic, priced both ways.

Concrete numbers beat adjectives. Take a mid-sized production service with three workloads: a user-facing assistant on Claude Sonnet 4.6 at 60M input and 6M output tokens a month, a reasoning-heavy feature on GPT-5.5 at 30M input and 3M output, and a bulk classification pipeline on DeepSeek V4 Pro at 200M input and 20M output. The table prices the same traffic at the providers' official list rates and at a discounted gateway's catalog rates. The token bill lands at $614.40 direct versus $501.96 through the gateway, a difference of $112.44 a month, roughly 18% on identical traffic. Whether that number matters depends on your scale, but note what it does not include: the engineering weeks of building and maintaining three separate integrations, which for most teams dwarfs the unit-price delta in the first year.

Where each path surprises teams.

The pattern across all of these is that the costs of direct integration are diffuse and show up as engineering time, while the costs of a gateway are concentrated and show up as architecture decisions. Diffuse costs are easy to underestimate at design time and hard to unwind later. Concentrated costs are at least visible on day one, which makes them easier to mitigate deliberately.

• The second provider doubles your integration surface, not your key count. New SDK, new rate-limit semantics, new streaming edge cases, new error formats, and a second status page on the on-call rotation.
• Retry logic tuned for one provider misbehaves on another. Aggressive retries that are safe against one vendor's 429s can amplify load during another vendor's overload incidents.
• Key sprawl is a governance problem before it is a security one: keys in several consoles, several rotation procedures, and no single answer to "which keys can spend money in prod right now."
• Direct integration gives you no unified spend view. Answering "what did feature X cost yesterday" across providers means building your own metering pipeline or waiting for month-end exports.
• Gateways surprise in the opposite direction: an extra hop adds latency (typically tens of milliseconds, small next to generation time, but measure your own p95), and the gateway becomes a critical-path dependency with its own failure modes.
• Cross-model failover is never free. Different model families produce different outputs, so a silent fallback can change product behavior without raising a single error. Log which model actually served every request.

Direct vs self-hosted gateway vs hosted gateway.

There are really three options, not two, because the gateway layer can live in your own cluster. Open-source gateways give you the unified interface with full control of the data path, at the price of operating another service. Hosted gateways remove the ops burden and often discount the tokens themselves, at the price of a third party in the request path. The table compares the three on the dimensions that decide real deployments.

A practical playbook, whichever way you go.

The first two items are the ones that make every later decision cheap. Teams that treat the endpoint and model as data can trial a gateway on one low-risk workload, measure latency and error rates against their direct baseline for a week, and adopt or revert with a config flip. Teams that hardcoded a provider SDK into fifty call sites cannot.

• Code against the OpenAI-compatible surface from day one, even when integrating directly. Nearly every provider and every gateway exposes it, and it is what keeps your switching cost at one config change instead of a rewrite.
• Make the base URL and model ID configuration, not code. Changing models or endpoints should never require a deploy, and staging should be able to point at a different endpoint than prod.
• Centralize keys in a secret manager with separate keys per environment and a scheduled rotation. A key that lives in three developers' shells is not a credential, it is a liability.
• Set per-key budgets and spend alerts before launch, not after the first surprising invoice. If your endpoint supports per-key limits, use them as a blast-radius control for leaked keys too.
• Decide failover semantics explicitly: retry the same model, fall back to a sibling model in the same family, or fail closed. Cross-family fallback changes output behavior, so it should be a product decision, not a default.
• If unit price is a primary driver, route through a discounted OpenAI-compatible endpoint. APIsRouter, for example, lists global models 20% below official list and Chinese models below their official rates, bills pay-as-you-go with no subscription, and adds +100% balance on the first top-up.
• Keep a direct-integration escape hatch. Nothing in your application code should assume the gateway exists; if it is ever down or you outgrow it, pointing the base URL back at a provider should be a five-minute change.

Migration in practice: one client, either backend.

Because the gateway speaks the same OpenAI-compatible protocol as the providers, the migration is not a rewrite. The same client object serves both paths, and the only things that change are the base URL, the key, and optionally the model ID. Tool calls, streaming, and JSON mode ride through unchanged. Before pointing production at any new endpoint, smoke-test the exact model IDs you plan to run with a one-off request, and confirm the model list matches what your config expects:

import os
from openai import OpenAI

# Direct or gateway is decided by config, not code.
# Direct:   LLM_BASE_URL=https://api.openai.com/v1
# Gateway:  LLM_BASE_URL=https://api.apisrouter.com/v1
client = OpenAI(
    base_url=os.environ["LLM_BASE_URL"],
    api_key=os.environ["LLM_API_KEY"],
)

resp = client.chat.completions.create(
    model=os.environ.get("LLM_MODEL", "claude-sonnet-4-6"),
    messages=[{"role": "user", "content": "Reply with one word: ok"}],
    max_tokens=16,
)
print(resp.choices[0].message.content)

FAQ