LLM usage tracking per feature: know what each feature actually costs

Quick answer: attribution is a logging problem, not a billing problem.

No LLM provider knows what a "feature" is in your product. The invoice knows models, keys, and dates. Your support chat, your nightly summarizer, and your autocomplete all collapse into the same line items unless something on your side stamps each request with where it came from. That stamp has to be applied at request time; there is no way to reconstruct it from the bill afterwards. The good news is that the raw material is already there. Every OpenAI-compatible chat completion response carries a usage object with exact prompt and completion token counts for that request. Multiply those by a price table and you have per-request cost to fractions of a cent. All that per-feature tracking adds is identity: which feature, which tenant, which attempt. In practice the setups that survive contact with production combine two layers. A coarse layer, one API key per feature, gives you a provider-side split that works even when your own logging breaks. A fine layer, a logging wrapper that every feature calls through, gives you per-request rows in your own database: feature, model, tokens, cost, latency, status. The rest of this page shows what each layer catches, what a real month looks like broken down this way, and the exact wrapper code.

Where the numbers come from: the usage object and the bill.

Start with what the API gives you for free. A non-streaming chat completion returns usage.prompt_tokens, usage.completion_tokens, and usage.total_tokens alongside the message. These are the same counts the provider bills against, so a log built from them reconciles against the invoice instead of estimating. Some providers also break out cached or reasoning tokens in a details sub-object; treat those fields as optional and log them when present. Streaming is the classic gap. A streamed response yields content chunks and, by default on many stacks, no token counts at all. The OpenAI-compatible fix is to send stream_options with include_usage set to true, which appends a final chunk carrying the usage object after the content finishes. Skip this and every streamed feature reports zero tokens in your logs while billing normally, which is exactly the kind of silent undercount that makes teams distrust their own dashboards. The bill side is coarser. Most provider consoles break spend down by model and by day, and some by key or project. That is enough to answer "how much did we spend on Sonnet yesterday" and useless for "what does the onboarding assistant cost per signup". Between the per-request usage object and the per-day invoice there is a resolution gap, and per-feature tracking is simply the discipline of filling it yourself. Once both sides exist, reconcile them on a schedule. Sum your logged cost per model per day and compare it to the provider's number for the same window. The two will never match to the cent, because clock boundaries and in-flight requests differ, but they should track within a percent or two. A growing gap means something specific: streamed calls logging zero, a code path bypassing the wrapper, or a price-table entry that drifted from the rate you are actually billed. Reconciliation turns each of those from a slow leak into a ticket.

A worked month: four features, one bill, no mystery.

Here is a mid-size product with four LLM-backed features, priced at real per-token rates. Before attribution, this team saw a single number climbing toward $1,600 a month and assumed the user-facing chat was responsible, because chat is what everyone watches. The split tells a different story. The nightly document summarizer, a background job nobody had looked at since it shipped, is about two thirds of the entire bill. Chat is roughly a quarter. The two highest-traffic features by request count, autocomplete and search rewriting, barely register in dollars because they run on budget models with short prompts. The follow-up move is only visible because the split exists: the summarizer's output is machine-consumed, so prose quality barely matters, and moving that one feature from gpt-5.4 to deepseek-v4-pro at $0.3915 and $0.783 per million tokens takes its line from $1,080 to about $164 and cuts the total bill by more than half. Nothing about the user-facing product changes.

Why per-feature spend surprises teams.

Each of these has the same shape: spend that is real on the invoice and invisible in the places engineers actually look. Attribution does not make any of them cheaper by itself. It makes them show up, which is the precondition for every fix in the rest of this page. There is also an organizational version of the same failure. When the bill is one number, it belongs to whoever pays it, usually nobody on the engineering side. When each feature has a line item, the team that owns the feature owns its cost, and a summarizer that quietly doubles gets the same treatment as a latency regression: noticed, assigned, fixed.

• Shared keys hide everything. When four features bill through one key, the provider dashboard is arithmetically incapable of splitting them, no matter how good the console is.
• Input tokens dominate and nobody watches them. Prompts, RAG context, and re-sent history are billed on every call; teams eyeball response length and miss the ten-times-larger input side.
• Background jobs have no user complaining about them. Cron summarizers, embedding refreshers, and eval runs burn steadily with zero product visibility.
• Retries multiply silently. A flaky upstream plus a three-attempt retry loop means some requests bill four times, and without an attempt counter in the logs that reads as organic growth.
• Free-text features drift. Cost per request tracks user input length, so the same feature gets more expensive as users paste bigger documents, with no code change to point at.
• Streaming without include_usage logs zero tokens. The feature looks free in your telemetry while billing normally on the invoice.

Key per feature, metadata tags, or middleware: what each layer catches.

There are three practical mechanisms for attaching identity to a request, and they are complements rather than competitors. The table compares them as they behave in production. The combination that works is keys plus middleware. Keys give you a provider-side number that keeps working when your logging pipeline is the thing that broke, and per-key quotas, where the provider supports them, turn a runaway feature into a capped feature instead of a surprise invoice. Middleware gives you the resolution keys cannot: per-tenant splits, latency, retry counts, and cost computed per request. Metadata fields are worth sending when your provider accepts them, but treat them as advisory; support and reporting vary enough across vendors that they should never be the only mechanism. A note on naming, because it decides whether the key layer survives growth. Name keys after the code path, not the person who created them or the quarter they were created in. A key called autocomplete-prod tells you what broke when it starts erroring and what leaked if it shows up somewhere it should not. A key called temp-test-2 tells you nothing, and every ledger that contains one eventually contains ten.

Seven habits that keep spend attributable.

The last two habits are where attribution pays for itself. The worked example above found more than $900 a month in a single routing change, and the budget alerts are what let you make that kind of change calmly instead of during an invoice emergency.

• Issue one API key per feature and name it after the feature. Rotation stays manageable when keys map one-to-one to code paths, and a leaked key implicates one feature instead of all of them.
• Log a usage row on every response: timestamp, feature, tenant, model, prompt tokens, completion tokens, computed cost, latency, and status. One table, one row per call.
• Compute cost at write time from a versioned price table. Prices change; recomputing history with current rates corrupts every trend line you have.
• Turn on stream_options include_usage for every streaming call, and count the final usage chunk, not the content chunks.
• Set per-feature monthly budgets and alert at 80% of budget, not at overage. An alert that fires after the money is spent is a report, not an alert.
• Log the attempt number on retries so a degraded upstream shows up as retry cost, distinct from organic traffic growth.
• Route each feature to the cheapest model that passes its own quality bar, and consider pointing the traffic at an OpenAI-compatible gateway with lower unit prices. One option is APIsRouter, which bills pay-as-you-go with no subscription and lists global models 20% below official rates, so the same per-feature ledger buys more headroom without touching the code above the base URL.

A tracked client: one choke point for every feature.

The implementation is deliberately boring: one function that every feature calls instead of constructing its own client. It selects the key for the feature, makes the call, reads the usage object, computes cost from the price table, and writes one row. Because it is a single choke point, adding a column later, say attempt number or tenant ID, is one edit rather than a hunt across the codebase. Point base_url wherever your traffic bills; the usage object and response shape are identical for any OpenAI-compatible endpoint, which is what makes the wrapper portable across providers and gateways. The SQL below it is the entire monthly report. Where the rows go matters less than that they exist. A Postgres table works for years at most products' volumes; a structured log line shipped to whatever you already run for observability works just as well. Once the rows accumulate, the derived numbers arrive almost for free: cost per signup for the onboarding flow, cost per seat for each tenant, cost per document for the summarizer. Those are the unit economics questions that finance eventually asks, and the teams that log per request answer them with a query instead of a guess.

import time
from openai import OpenAI

# USD per 1M tokens. Versioned in git; never edit history.
PRICES = {
    "claude-sonnet-4-6": (2.40, 12.00),
    "gpt-5.4": (2.00, 12.00),
    "deepseek-v4-flash": (0.126, 0.252),
    "deepseek-v4-pro": (0.3915, 0.783),
}

KEYS = {  # one key per feature
    "support_chat": "sk-...chat",
    "doc_summarizer": "sk-...sum",
    "autocomplete": "sk-...ac",
}

def tracked_chat(feature: str, model: str, messages: list, **kw):
    client = OpenAI(
        base_url="https://api.apisrouter.com/v1",
        api_key=KEYS[feature],
    )
    t0 = time.monotonic()
    resp = client.chat.completions.create(
        model=model, messages=messages, **kw
    )
    p_in, p_out = PRICES[model]
    u = resp.usage
    cost = (u.prompt_tokens * p_in + u.completion_tokens * p_out) / 1_000_000
    log_row(  # your DB insert or structured log line
        feature=feature,
        model=model,
        tokens_in=u.prompt_tokens,
        tokens_out=u.completion_tokens,
        cost_usd=round(cost, 6),
        latency_ms=int((time.monotonic() - t0) * 1000),
    )
    return resp

FAQ