LLM cost monitoring and observability for production traffic

Quick answer: meter every request yourself.

The unit of LLM cost is the individual request, and every OpenAI-compatible response already carries the numbers you need: the usage object reports prompt_tokens and completion_tokens. Wrap your client once, capture those two integers plus the model ID and a feature tag, and you have the raw feed for every dashboard, budget, and alert you will ever build. Everything else in LLM cost observability is aggregation on top of that one log line. Do the pricing at ingest, not at query time. Multiply tokens by a rate table you version in git and store the computed dollar figure next to the raw counts. Prices change, models get remapped, and a query-time join against current prices silently rewrites history. A stored cost_usd field survives every repricing and makes month-over-month comparisons honest. Then alert on how fast spend moves, not on where it ends up. A monthly budget alarm fires after the damage is done. Hourly per-feature spend compared against a trailing baseline catches a runaway retry loop within the hour, which is the difference between a strange blip on a graph and a four-figure incident review.

Where the money goes, and the fields that explain it.

Input tokens deserve special attention. A chat product re-sends the conversation on every turn, a RAG pipeline attaches thousands of tokens of retrieved context per query, and an agent re-sends its entire growing window dozens of times per task. In all three shapes, input volume grows quadratically with conversation length while output stays flat, which is why mature LLM bills skew heavily toward input even though output tokens carry the higher unit price. Cached input is the one nuance worth encoding early. Several providers discount repeated prompt prefixes, and the discount rules differ per provider and per model. When the API returns cached token counts, record them as their own field. Do not bake an assumed cache discount into your price table unless you have verified it against a real invoice, because a silently broken cache is one of the most common sources of unexplained spend growth.

• model: unit prices differ by more than 10x between flagship and budget tiers, so a routing change moves the bill more than any traffic change.
• prompt_tokens: the dominant cost in RAG and agentic traffic, because system prompts, retrieved documents, and chat history are re-sent as input on every call.
• completion_tokens: billed at roughly 2x to 6x the input rate depending on the model, but usually a small share of volume outside long-form generation.
• feature and tenant tags: without a per-feature or per-customer dimension in the log, spend can only ever be explained at account level.
• latency and status: retries and timeouts correlate directly with double billing, and you cannot see that unless request outcomes live in the same record as token counts.

A worked example: one product, four features, one bill.

Here is a realistic mid-size deployment: a SaaS product running support chat, RAG answers over customer documents, nightly batch summarization, and a small agentic workflow feature. Each feature carries its own model, traffic profile, and token shape. The table prices a typical day at the catalog rates used later on this page. The lesson is in the ratios. The agent feature is under 1 percent of requests but about 44 percent of daily spend, because each run re-sends a 40,000-token window. Support chat produces the most requests and looks dominant on any request-count dashboard, yet lands in the middle of the cost table. And 100,000 batch summaries on a budget model cost less than twenty-nine dollars, which is why token-weighted attribution, not request counting, is the only honest way to read an LLM bill.

Why LLM bills surprise teams that already have monitoring.

The common thread is that conventional APM answers "is it up and fast" while LLM spend fails along a different axis: the same endpoint, same latency, same status codes, and a bill that tripled. Cost has to be treated as a first-class signal with its own baselines and its own alerts, computed from your own logs rather than reconstructed from an invoice three weeks later.

• Retries can bill twice: a request that times out after generation started still consumed tokens, and the retry pays full price again. Unbounded retry loops are the classic overnight-spike incident.
• Prompt growth is fleet-wide: adding 2,000 tokens to a shared system prompt raises the cost of every call from every user the moment it deploys, with zero change in traffic graphs.
• Agentic loops decouple requests from tokens: request counts stay flat while tokens per request climb as context accumulates, so request-based dashboards show nothing.
• Fallback routing upgrades price silently: a failover from a budget model to a flagship keeps availability green while multiplying unit cost.
• Cache regressions are invisible: reordering a prompt prefix or injecting a timestamp near the top can disable provider prompt caching without changing a single output.
• Provider dashboards lag: aggregates typically update hours behind real traffic, so a runaway loop burns budget long before the official graph moves.
• Averages hide the tail: mean cost per request looks stable while a handful of long-context users or documents dominate spend. Attribution needs a P95 view, not a mean.

Four ways to get the data, compared.

There are four practical architectures for LLM cost observability, and they are not mutually exclusive. Most teams start with a client wrapper because it is a day of work and owns the raw data forever, then add a tracing platform when prompt iteration and evaluation become the bottleneck, and consolidate metering at a gateway once traffic spans several model vendors. Whatever you add later, keep the wrapper. Platforms and gateways come and go, but a JSONL file of request-level token counts in your own storage is the one asset that makes every future migration, audit, and repricing painless.

Practical fixes: the checklist that keeps spend flat.

The last two levers compound: moving a feature from a flagship to a mid-tier model cuts its cost several-fold, and billing those tokens at below-list rates cuts what remains. One example of the latter is APIsRouter, an OpenAI-compatible gateway with pay-as-you-go billing and no subscription, where global models are priced 20% below official list, Chinese models sit below their official rates, the first top-up adds +100% balance, and the checkout takes payment first and emails the key with no signup. Because it speaks the same protocol, the wrapper from the next section works unchanged. The ladder below shows what routing tiers look like at gateway catalog rates. The worked example above already uses this pattern: batch work on a budget model, user-facing generation on a mid-tier, frontier reasoning reserved for the steps that need it.

• Wrap the client once. Every model call in the codebase goes through one function that records model, token counts, computed cost, latency, and a feature tag. Ban direct SDK calls in code review.
• Version the price table in git and stamp cost_usd at ingest, so historical spend never changes when prices do.
• Issue one API key per service or feature. Per-key breakdowns then come free on whatever billing dashboard sits upstream, and revoking a leaked key kills one feature instead of production.
• Set two alerts: hourly per-feature spend against a trailing seven-day baseline for anomalies, and a hard daily ceiling per key as the backstop.
• Cap max_tokens per route and give every integration a retry budget with exponential backoff. Most catastrophic bills are a retry loop, not organic growth.
• Sample verbose payload logging at around 1 percent for debugging, but record token counts and cost on 100 percent of requests. Counts are cheap; payloads are not.
• Walk traffic down the price ladder, and consider routing through a cheaper OpenAI-compatible endpoint so the unit price drops without touching the instrumentation.

Instrumentation you can paste today.

The wrapper below is the whole monitoring core: one function, one structured log line per request, cost computed at ingest from a versioned price table. Ship the lines to stdout, a JSONL file, or your metrics pipeline; the fields stay the same either way. Because the endpoint is OpenAI-compatible, the same code meters every model in the routing ladder above. The second snippet is the aggregation you will run most often: daily spend grouped by feature, straight off the log file. The same one-liner adapted per model or per key answers almost every question a finance or capacity review will ask.

import json
import time
from openai import OpenAI

# USD per 1M tokens. Version this table in git; never edit history.
PRICES = {
    "claude-sonnet-4-6": (2.40, 12.00),
    "gpt-5.4-mini": (0.60, 3.60),
    "deepseek-v4-flash": (0.126, 0.252),
}

client = OpenAI(base_url="https://api.apisrouter.com/v1", api_key="sk-...")

def tracked_completion(feature: str, **kwargs):
    t0 = time.monotonic()
    resp = client.chat.completions.create(**kwargs)
    p_in, p_out = PRICES[kwargs["model"]]
    usage = resp.usage
    cost = (usage.prompt_tokens * p_in + usage.completion_tokens * p_out) / 1_000_000
    log_line = {
        "ts": time.time(),
        "feature": feature,
        "model": kwargs["model"],
        "in_tokens": usage.prompt_tokens,
        "out_tokens": usage.completion_tokens,
        "cost_usd": round(cost, 6),
        "latency_ms": round((time.monotonic() - t0) * 1000),
    }
    with open("llm-usage.jsonl", "a") as f:
        f.write(json.dumps(log_line) + "\n")
    return resp

FAQ