Run LlamaIndex on a custom OpenAI-compatible endpoint.

Quick answer: OpenAILike with api_base and is_chat_model=True.

LlamaIndex's OpenAI class validates model names against OpenAI's catalog, so it rejects ids it does not recognize. The framework's answer is OpenAILike, a separate class in the llama-index-llms-openai-like package that skips validation and speaks the same protocol: set api_base to https://api.apisrouter.com/v1, pass your key and any served model id, and set is_chat_model=True so requests go to /chat/completions. That last flag is load-bearing. OpenAILike cannot know whether an unfamiliar id is a chat model, and without the flag it treats the model as a legacy completions model and posts to an endpoint the gateway does not serve for this purpose. Setting it explicitly is the difference between a working setup and a 404 that looks like a routing bug.

import os
from llama_index.llms.openai_like import OpenAILike

llm = OpenAILike(
    model="claude-sonnet-4-6",          # any catalog id
    api_base="https://api.apisrouter.com/v1",
    api_key=os.environ["APISROUTER_API_KEY"],
    is_chat_model=True,                  # required: chat, not completions
    is_function_calling_model=True,      # enables agent/tool use paths
    context_window=200000,
)

print(llm.complete("Summarize retrieval-augmented generation."))

Where the LLM sits in a LlamaIndex pipeline.

LlamaIndex (run-llama on GitHub, roughly 51K stars) is a data framework: documents get chunked and embedded into an index, queries retrieve relevant chunks by vector similarity, and only then does the LLM enter, synthesizing an answer from the retrieved context. Agents add a second LLM duty, deciding which tools and query engines to call. The division of labor matters for routing. Retrieval runs on the embedding model; synthesis and agency run on the LLM. OpenAILike moves the second workload to the gateway while the embedding model, and therefore every vector already in your index, stays exactly where it is. Repointing embeddings is a re-index; repointing the synthesizer is a constructor argument. Wire it in either globally through Settings.llm, which every query engine and agent built afterwards inherits, or per component for pipelines that mix models. The model field passes through as a plain string, so the synthesizer can be a Claude id while a summarize-then-index preprocessing step runs on a fast DeepSeek id, both through one endpoint and key.

Full setup: RAG query engine, and the TS variant.

The RAG pattern below sets the gateway model globally and leaves the embedding configuration alone. context_window deserves attention: LlamaIndex plans how much retrieved text to pack per synthesis call against this figure, and OpenAILike defaults it conservatively for unknown models. Declaring the model's real window means fewer refine passes over the same nodes, which is both quality and cost. In TypeScript, LlamaIndex.TS reaches the same endpoint through its OpenAI class, which accepts baseURL alongside apiKey and model. There is no separate OpenAILike class in the TS package; the baseURL override is the documented route.

import os
from llama_index.core import Settings, VectorStoreIndex, SimpleDirectoryReader
from llama_index.llms.openai_like import OpenAILike

Settings.llm = OpenAILike(
    model="claude-sonnet-4-6",
    api_base="https://api.apisrouter.com/v1",
    api_key=os.environ["APISROUTER_API_KEY"],
    is_chat_model=True,
    is_function_calling_model=True,
    context_window=200000,
)
# Settings.embed_model unchanged: index and vectors stay put

docs = SimpleDirectoryReader("./reports").load_data()
index = VectorStoreIndex.from_documents(docs)

engine = index.as_query_engine(similarity_top_k=5)
print(engine.query("What drove margin change quarter over quarter?"))

Choosing the synthesis model.

RAG makes model comparison unusually rigorous: hold the index and a query set fixed, swap the synthesizer id, and diff the answers against sources. Behind one endpoint each candidate is a one-line change, and the per-key usage view prices each run of the evaluation set.

• Synthesis quality is answer quality: the model reads retrieved chunks and writes the response, so faithfulness to sources is the trait to optimize. claude-sonnet-4-6 is the dependable middle; gpt-5.4 is a strong second opinion.
• High-volume query workloads, internal search boxes, batch document QA, put most calls through the synthesizer; claude-haiku-4-5-20251001 keeps per-query cost flat where questions are routine.
• Agent workloads (query engines as tools, multi-step reasoning) need is_function_calling_model=True and an id with dependable tool use; this is where stronger models pay for themselves.
• Long retrieved contexts, high top-k or big chunks, favor gemini-3.1-pro-preview and a truthfully declared context_window, trading fewer refine passes for a larger single call.

Pay-as-you-go · transparent per-model pricing

Selected models are priced below official list prices. Exact input, output, cache, and per-request prices are shown for each model.

Failure modes and version pinning.

Forgetting is_chat_model. The default treats an unknown id as a completions-era model and posts to the legacy endpoint; the result is a route-shaped failure on a config that otherwise looks right. This is the signature OpenAILike mistake, worth checking before anything else. Using the plain OpenAI class with a non-OpenAI id. That class validates model names and raises on ids it does not know, before any request is sent. If you see an unknown-model error with no network traffic, you are in the wrong class; switch to OpenAILike. Leaving context_window at its default. Synthesis planning packs retrieved text against this figure. A conservative default on a 200k model means more, smaller synthesis calls and more refine passes over your nodes; declare the real window and response synthesis gets both better and less chatty. Package layout since v0.10. LlamaIndex split into namespaced packages; OpenAILike lives in llama-index-llms-openai-like and is imported from llama_index.llms.openai_like. Old imports from the monolithic llama_index package predate the split. Pin the specific integration package in requirements, not just llama-index-core. Agent paths silently degrade without the function-calling flag. is_function_calling_model=False routes agents through fallback prompting instead of native tool calls, which works but is measurably less dependable. For tool-capable ids like those in the table, set it to True explicitly. Embeddings are a different migration. Settings.embed_model and the LLM are independent; moving synthesis does not invalidate vectors, but repointing the embedder means re-embedding the corpus. Treat them as separate decisions on separate timelines.

Who routes LlamaIndex through a gateway.

• RAG builders who want Claude-quality synthesis over an existing index without touching the embedding side or re-indexing anything.
• Teams running document QA at volume, where a fast synthesizer id changes the unit economics of every query and per-key usage makes cost per thousand queries visible.
• Engineers evaluating synthesis models against a fixed index and query set, one constructor string per candidate.
• Agent builders using query engines as tools, who want dependable function calling and a single billing surface across every model in the pipeline.
• Developers without access to a given vendor's billing. Top-up based access with no card requirement removes the sign-up dependency per provider.

Verify the endpoint and debug the first query.

List the gateway's models first; the model argument must match a served id exactly, version suffix included. First-query failures follow a pattern. A route-shaped 404 means is_chat_model was left unset. An unknown-model error raised locally with no request sent means the plain OpenAI class is in use instead of OpenAILike. A 401 means the key is wrong for the endpoint, or an environment fallback resolved differently than the constructor arguments you intended; print the effective values. A model-not-found error from the gateway is an id typo. Once queries flow, the APIsRouter console shows per-request model, token counts, and spend. For RAG the shape to watch is tokens per query: retrieved context dominates input tokens, so top-k and chunk-size tuning show up in the usage view as directly as model choice does.

curl -s https://api.apisrouter.com/v1/models \
  -H "Authorization: Bearer $APISROUTER_API_KEY" | head -50

FAQ