Cost per user: allocation and billing logic

Offering generative AI functionality inside a SaaS application transforms the cost structure of software radically. Where traditional web applications scale with predictable margins on compute and storage, every interaction with a Large Language Model (LLM) carries a direct, variable purchase cost. When users can run prompts, analyze documents, or start agent loops without limits, a lack of fine-grained attribution inevitably leads to margin pressure or even loss-making customers. A flat-rate subscription without underlying usage accounting is financially untenable as soon as power users consume hundreds of megatokens per month.

To guarantee a healthy gross margin, a robust architecture for cost attribution and billing logic is essential. It requires that every request is not only handled technically but also labeled, synchronously or asynchronously, with metadata about the actor, organization, session, and task context. In this article we analyze how to set up a reliable metering pipeline, deal with asymmetric token prices, process partial streams, and turn usage data into clean invoice lines without operational overhead.

The architecture of a metering pipeline

Within pillar A3 (cost & usage accounting), accurate recording forms the foundation of any cost policy. See the anchor article on managing rate limits, tokens and costs for the basic principles around throughput rates and global budget ceilings. Where global limits protect the total infrastructure against uncontrolled spending, a metering pipeline splits the incoming payload and the outgoing response at the level of the individual tenant.

In a scalable architecture, cost recording must not increase the latency of the model request. A synchronous database write in the critical path of the HTTP response introduces needless delay and creates a single point of failure. That is why we split metering into two phases: runtime payload tagging and asynchronous reconciliation through an event bus.

The gateway captures the raw response from the model provider, including the header or body information about exact token usage (prompt tokens, completion tokens, and any cached tokens). The gateway then generates an immutable billing event (usage event) containing at least the following attributes:

Differentiated token prices and composite costs

A common mistake in early billing systems is calculating with an average token price. Modern models apply strongly asymmetric rates: output tokens are usually three to four times as expensive as input tokens. On top of that, virtually all leading providers offer discounts (ranging from 50% to as much as 90%) on input tokens when these are reused through prompt caching, although some providers do charge storage costs per hour for explicit caching. If your application repeatedly submits long system prompts or document contexts, the purchase price drops dramatically.

To calculate margins cleanly, the billing engine has to apply a dynamic price table that stays historically accurate. If a provider lowers its rates on the fifteenth of the month, events from before that date must be calculated at the old rate and later events at the new one. In code we express costs in micro-units (for instance in ten-thousandths of a eurocent or in microdollars) to avoid rounding errors across millions of requests.

// Voorbeeld van een cost calculation routine met prompt caching en reasoning tokens
interface TokenUsage {
  promptTokens: number;
  cachedPromptTokens: number;
  completionTokens: number;
  reasoningTokens?: number;
}

interface ModelPricing {
  effectiveDate: string; // ISO 8601
  inputCostPerMillion: number;
  cachedInputCostPerMillion: number;
  outputCostPerMillion: number;
}

function calculateRawCostMicroUSD(usage: TokenUsage, pricing: ModelPricing): number {
  const nonCachedPrompt = usage.promptTokens - usage.cachedPromptTokens;
  
  const inputCost = (nonCachedPrompt / 1_000_000) * pricing.inputCostPerMillion;
  const cachedCost = (usage.cachedPromptTokens / 1_000_000) * pricing.cachedInputCostPerMillion;
  const outputCost = (usage.completionTokens / 1_000_000) * pricing.outputCostPerMillion;
  
  const totalUSD = inputCost + cachedCost + outputCost;
  return Math.round(totalUSD * 1_000_000); // Retourneert kosten in micro-USD
}

When a model generates so-called reasoning tokens (as with the OpenAI o1 and o3 series), the provider bills these as output tokens, even though the end user often never sees these thinking steps directly in the interface. The metering pipeline must record these hidden tokens in completion_tokens to avoid unexplained margin spikes.

Aggregators, gateways, and infrastructural overhead

Many organizations choose not to talk to individual model providers directly but to use an abstraction layer. Read the article on how an LLM API aggregator works to understand how routing and fallback between providers are arranged across the board. An aggregator does, however, introduce an extra financial dimension: the markup or surcharge per transaction.

When you use a managed aggregator (such as OpenRouter, Helicone, or Portkey), you pay a platform margin or a fixed amount per million tokens on top of the bare token price. If you host your own open-source gateway (such as LiteLLM or a custom proxy), there are fixed hosting costs for containers, load balancers, and caching layers. In a mature attribution model you reserve an adjustable overhead factor (for instance 5% to 12%) on top of the raw purchase costs to spread the operational infrastructure fairly across active users.

The attribution formula for the eventual cost price per request $K_{totaal}$ then looks like this:

K_totaal = (K_tokens_ruw * (1 + M_infra)) + K_tooling + K_retrieval

Here M_infra stands for the infrastructural gateway margin, K_tooling for the cost of external API calls within function calling (such as web scraping or code execution), and K_retrieval for the embedding and vector search costs incurred ahead of the prompt.

Streaming responses and dropped connections

One of the most stubborn problems in LLM billing arises with Server-Sent Events (SSE) and streaming answers. When a user generates an answer and closes the page halfway through or clicks 'Stop generating', the client drops the HTTP connection. The model provider on the back end, however, often keeps generating dozens more tokens in the background before the TCP cancel signal is processed, or bills the tokens up to the exact moment of cancellation.

Two concrete failure modes occur regularly here:

• No final payload received: On an aborted stream the provider often no longer sends the closing JSON chunk with the definitive usagestatistics. The application is left with an incomplete number.
• Client-side token counting diverges: The number of tokens the client actually displayed is lower than the number the server produced and the provider charges for.

To solve this, the proxy gateway has to track the outgoing SSE chunks locally. When the stream stops abruptly, the gateway calculates a fallback estimate by tokenizing the number of words and characters received through a fast local tokenizer (such as tiktoken). As soon as the provider later makes the real figures available through a webhook or usage report, the billing system runs an automatic reconciliation.

Key management, tenant authentication, and metering data security

Watertight billing stands or falls with reliable authentication at the gate. Consult the guide on managing LLM API keys securely for insight into rotation mechanisms and zero-trust configurations. When a tenant creates API keys for its own staff or automated pipelines, every key has to be immutably tied to a tenant ID, a cost center, and optional sub-labels.

To keep bad actors or compromised applications from manipulating billing data, the client must never report token usage itself. Attribution happens exclusively server-side on the basis of cryptographically validated sessions. Below is a schematic representation of the request flow and measurement points:

[Client / Frontend]
        │
        ▼ (1) Request met Tenant API-Key
[API Gateway & Auth Proxy] ──► Valideer tenant status & prepaid tegoed
        │
        ▼ (2) Forward met Master Provider-Key + Request-ID
[LLM Provider (OpenAI/Anthropic)]
        │
        ▲ (3) Response stream + Provider Usage Metadata
[API Gateway] ──► (4) Genereer Usage Event naar Message Queue
        │
        ▼ (5) Stream doorgeven naar Client
[Client]

[Message Queue Consumer] ──► (6) Bereken kosten & update Saldo/Factuurregel

Billing models in practice: credits, pay-as-you-go, and hybrid tiers

Once accurate metering data comes in, it has to be translated into a commercial model. In B2B SaaS we see three common methods for passing LLM costs on to customers:

The hybrid model calls for strict threshold monitoring. When a user reaches 80% of their monthly quota, the system sends automated notifications to the administrator. If 100% is exceeded, the account can switch automatically to overage rates or temporarily downgrade to a lighter model.

Currency fluctuations and invoice reconciliation

Virtually all major LLM providers bill their services in US dollars (USD), while European SaaS companies invoice their customers in euros (EUR). That introduces currency risk. If the exchange rate moves between the moment of consumption and the moment of the monthly provider charge, the calculated margin can evaporate.

A professional billing engine therefore keeps two currency amounts per event: the purchase value in the original currency (USD) and the converted value in the functional currency (EUR) based on the daily rate (for instance through the European Central Bank API). At the end of the billing period a reconciliation round takes place:

1. The provider's monthly invoice is imported (through API or CSV export).
2. The total provider costs per model are compared with the sum of all recorded usage events in the database.
3. Any deviations (caused by delayed webhooks, rounding differences, or network errors) are analyzed. A deviation under 0.5% is classified as an acceptable rounding tolerance; larger deviations point to unrecorded requests in the proxy.

The hidden costs of quality control and fact-checking

In advanced AI applications, usage is rarely limited to a single prompt. To generate reliable answers, modern systems deploy verification passes, rerankers, or fact-checking mechanisms. See the article on fact-checking AI answers systematically for methods that detect hallucinations by means of secondary model calls.

Every validation step — such as a critic model that tests the generated answer against source documents — doubles or triples the total token usage of that single user action. If these secondary costs are not explicitly tied to the user's original request_id , blind spots appear in financial reporting. Management sees high central API costs while the individual user tables report a low cost price. Always assign an overarching trace_id that bundles all underlying sub-calls under the same user transaction.

Failure modes, auditing, and mitigation in billing conflicts

In production, incidents that put billing logic to the test occur sooner or later. Below we describe the three most common failure modes and the corresponding mitigation measures.

1. The infinite agent loop

Failure mode: An autonomous agent gets tangled in a logical error and fires thousands of calls with enormous payload sizes in a short time. Within a few hours thousands of euros' worth of tokens are consumed without the end user receiving a usable result.

Detection: Real-time anomaly detection at tenant level that raises an alarm as soon as token usage per minute deviates by more than 400% from the moving average.

Mitigation: Hard loop counters (a maximum of 10 iterations per agent task) combined with an automatic circuit breaker in the gateway that pauses the key for the specific session.

2. Double billing through automatic retries

Failure mode: A provider returns an HTTP 504 Gateway Timeout but has already processed and billed the tokens internally. The application performs an automatic retry and completes the task successfully after all. Without correlated logging, the customer is charged twice for one task.

Detection: Tie idempotency keys to all retries and compare them with the provider's usage logs.

Mitigation: Record failed attempts under an internal cost item 'infrastructure errors' rather than on the end user's invoice. Apply a commercial correction when the fault lay outside the customer's control.

3. Billing disputes over invisible tokens

Failure mode: A customer disputes an invoice because their staff asked short questions while the invoice lists enormous numbers of prompt tokens caused by huge RAG contexts or embedded PDF documents.

Detection: Reports that show only total costs without a breakdown into document processing versus chat interaction.

Mitigation: Offer customers a transparent observability dashboard showing per transaction how many tokens went into document context, system prompts, and the final answer. Transparency prevents receivables conflicts and encourages customers to structure their documents more efficiently.

Conclusion

Cost attribution in LLM applications is not an administrative side issue but a core component of software architecture in the AI era. By implementing a decoupled metering pipeline, tracking differentiated prices accurately, absorbing streaming interruptions, and passing overhead on fairly, you keep full control of your application's gross margins. A robust system protects both the SaaS company against unforeseen provider costs and the customer against unexplained invoices.