Run paper-qa against a custom OpenAI-compatible endpoint.

Quick answer: a router dict with api_base, reused per slot.

paper-qa's Settings object takes a model name plus an optional LiteLLM router config per slot. The router config is a model_list whose litellm_params carry api_base and api_key, which is the same documented pattern the README uses for locally hosted OpenAI-compatible servers; a gateway is simply that pattern with a public URL and a real key. Set llm and summary_llm to the model_name you declared, attach the config to both slots, and paper-qa routes through the gateway. The model string inside litellm_params keeps litellm's provider convention: openai/<id> tells litellm to speak chat-completions to your api_base, and the id after the slash is passed through to the endpoint, so Claude, GPT, Gemini, and GLM ids are all addressable with the same dict.

gateway_config = dict(
    model_list=[
        dict(
            model_name="claude-sonnet-4-6",
            litellm_params=dict(
                model="openai/claude-sonnet-4-6",
                api_base="https://api.apisrouter.com/v1",
                api_key=os.getenv("APISROUTER_API_KEY"),
                temperature=0.1,
            ),
        )
    ]
)

Where paper-qa spends tokens: three slots plus embeddings.

paper-qa (Future-House on GitHub, roughly 9K stars) does retrieval-augmented question answering over scientific PDFs with an agentic loop on top: an agent decides when to search your library, gathers evidence chunks, summarizes their relevance, and composes a cited answer. That maps onto three separately configurable LLM slots. summary_llm evaluates and condenses evidence per retrieved chunk, which makes it the volume slot. llm writes the final answer from the assembled evidence, the quality-critical step. And agent_llm (inside the agent settings) makes the tool-selection decisions that steer the loop. All three default to an OpenAI model, and each has a matching _config field (llm_config, summary_llm_config, agent_llm_config) that accepts the same router dict, so one gateway config object can be attached to each slot while the model name per slot stays independent. A common split is a fast id summarizing evidence and a frontier id writing answers, both through one endpoint and key. Embeddings are the fourth workload and deliberately separate: the embedding setting (default text-embedding-3-small) builds the vector index of your papers. Moving chat slots to a gateway does not move embeddings, and paper-qa supports local sentence-transformers (the st- prefix, via the local extras) if you want the index fully independent of any remote endpoint.

Full setup: Settings with per-slot configs.

The full pattern declares one router entry per model you want addressable and attaches configs slot by slot. Declaring two entries, a fast one for summaries and a strong one for answers, keeps the whole setup in one dict. The same routing works from the CLI, since pqa exposes the settings surface, but the Python path is the reproducible one for research use: the Settings object that produced an answer can be logged next to the answer itself.

import os
from paperqa import Settings, ask
from paperqa.settings import AgentSettings

def entry(model_id, **params):
    return dict(
        model_name=model_id,
        litellm_params=dict(
            model=f"openai/{model_id}",
            api_base="https://api.apisrouter.com/v1",
            api_key=os.getenv("APISROUTER_API_KEY"),
            **params,
        ),
    )

gateway = dict(model_list=[
    entry("claude-sonnet-4-6", temperature=0.1),
    entry("claude-haiku-4-5-20251001", temperature=0.1),
])

answer = ask(
    "What is the evidence for LK-99 room-temperature superconductivity?",
    settings=Settings(
        llm="claude-sonnet-4-6",
        llm_config=gateway,
        summary_llm="claude-haiku-4-5-20251001",
        summary_llm_config=gateway,
        agent=AgentSettings(
            agent_llm="claude-sonnet-4-6",
            agent_llm_config=gateway,
        ),
        paper_directory="./papers",
    ),
)

Choosing models per slot.

Tune with the evidence pipeline fixed: same library, same questions, swap one slot at a time. Behind one endpoint each candidate is a model_name string, and the per-key usage log prices each configuration per question, which is the number a lab actually budgets on.

• summary_llm runs once per evidence chunk, every question. On a serious library this is the overwhelming majority of calls, so a fast id (claude-haiku-4-5-20251001) sets the cost floor for the whole system while only having to judge relevance, not write prose.
• llm composes the cited answer from assembled evidence. This is where hedged, precise scientific writing either happens or does not; claude-sonnet-4-6 and gpt-5.5 are the dependable choices, and the slot is few calls per question so the premium is bounded.
• agent_llm steers the loop: whether to search again, gather more evidence, or answer. Weak decisions here waste tokens everywhere else, which makes a mid-tier or better id the economical choice despite the slot's low volume.
• Long-context ids like gemini-3.1-pro-preview are worth testing as the answer slot when questions pull evidence from many papers at once.

Pay-as-you-go · transparent per-model pricing

Selected models are priced below official list prices. Exact input, output, cache, and per-request prices are shown for each model.

The failure modes specific to paper-qa.

A slot left on its default. Setting llm and llm_config but not summary_llm_config leaves summarization on the default OpenAI model, which then demands OPENAI_API_KEY and fails (or silently splits your routing across two endpoints if that key exists). Each slot has its own _config field; attach the gateway dict to every slot you intend to move, agent_llm_config included. Names that do not line up. Settings.llm must equal a model_name in the model_list; litellm_params.model is what actually goes to the wire. Mismatch the outer name and the router has no route; typo the inner id and the gateway returns model-not-found. When debugging, check the two strings separately because they fail differently. Embeddings assumed to follow. The embedding slot builds and queries the vector index and has its own default and config. If you do not have an OpenAI key for the default embedding, configure embedding explicitly, or use local sentence-transformers via the st- prefix. Re-pointing embeddings later also means re-indexing: vectors from different embedding models do not mix. Missing generation limits for long answers. litellm_params accepts max_tokens per entry, and the local-endpoint examples upstream set it deliberately. An answer slot without a sensible limit can truncate long cited answers, which presents as model weakness but is a parameter. Blaming routing for parsing problems. paper-qa's quality depends on PDF parsing and chunking before any model sees text. If answers cite nothing on a library you know is relevant, inspect the indexing step; the gateway only sees what retrieval sends it.

Who routes paper-qa through a gateway.

• Research groups running literature QA over shared libraries, where per-key usage turns "what does the lab spend per question" from a guess into a report.
• Teams that want Claude-quality scientific writing in the answer slot while keeping the summarization volume on a fast id, one key for both.
• Builders embedding paper-qa in internal tools, replacing a bundle of vendor secrets with one gateway credential per environment.
• Benchmarkers comparing answer models on fixed evidence pipelines, where each candidate is a config string rather than a vendor integration.
• Developers without access to a given vendor's billing. Top-up based access with no card requirement removes the per-provider sign-up dependency.

Verify the endpoint and debug the first question.

Confirm the gateway serves the ids you declared; the litellm_params.model string after openai/ must match a served id exactly. The failure ladder on a first ask(): an error demanding OPENAI_API_KEY means some slot is still on its default model with no config attached; find which of llm, summary_llm, and agent_llm you did not move. A 401 from the gateway is the api_key inside litellm_params. A router error about an unknown model means Settings.llm does not match any model_name in the list. Failures during indexing rather than answering point at the embedding setting or PDF parsing, not chat routing. One question fans out into many summary calls plus agent steps plus the final answer, so after the first successful run, the APIsRouter console's per-request view shows the slot split in real tokens. That is the number to watch as the library grows, because summary volume scales with retrieved evidence, not with question count alone.

curl -s https://api.apisrouter.com/v1/models \
  -H "Authorization: Bearer $APISROUTER_API_KEY" | head -50

FAQ