Observability and Logging for LLM Applications

Published in the category Monitoring & Architecture

If you build a traditional REST API, observability is relatively straightforward. You log the endpoint, the HTTP status code (usually 200 or 500), and the response time. With Large Language Models (LLMs), it is best to throw this classic approach overboard immediately. An LLM API is non-deterministic. A request can take 50 milliseconds and generate 2 tokens one second, and five minutes later, with the exact same input, take 30 seconds and return 4000 tokens.

Without a well-thought-out observability strategy, you are flying blind. You cannot explain cost spikes, you do not know why an application suddenly feels slow to the end user, and you have no idea how often the model hallucinates or gets stuck in a loop. In this article, we discuss exactly what you need to log per call, how to handle strict privacy requirements around prompts, how to trace complex multi-step processes, and which alerts truly add value for your development and operations teams.

The Basics: What should you log per LLM API call?

To set up a robust monitoring system, you need structured logs (preferably in JSON) for every interaction with the LLM. This allows you to perform aggregations and breakdowns later. We categorize the required data points into four domains.

1. Configuration and Context

Models change under the hood. An update from the provider can cause your perfectly tuned prompt to suddenly deliver suboptimal results. Therefore, always log:

2. Performance and Latency

With LLMs, especially when streaming is used, classic Total Latency is not sufficient to measure the user experience. You need granular metrics:

3. Tokens and Costs

Tokens are the currency of LLMs. You want to know at any moment how much an interaction costs. For a comprehensive guide on how to use this data for financial management, read our article on monitoring costs.

4. Tracing and Identifiers

Errors and bottlenecks always occur. To debug these, you need strong identifiers. Especially when dealing with HTTP 429 (Too Many Requests), as described in our article on rate limits and costs.

Privacy and Security: Why you shouldn't just save prompts

It is tempting to simply store the full incoming prompt and the outgoing response in your logging system (such as Elasticsearch, Datadog, or Splunk) for easy debugging later. However, in many cases, this is a huge security risk and a violation of the GDPR.

Users enter unpredictable, unstructured data. This can contain medical information, financial data, or trade secrets (PII - Personally Identifiable Information). If this data ends up unencrypted in your centralized logs, you immediately create a massive data breach risk. Developers with access to the logs would suddenly have access to sensitive personal data. For more context on secure architectures, you can consult our external guide on privacy by design for LLMs.

How do you solve this?

  1. Hashing: As mentioned earlier, use a hash of the prompt. For analytical purposes, you often only want to know if prompt A is used more often than prompt B. A hash is sufficient for that.
  2. Data Masking / PII Scrubbing: Use libraries (such as Microsoft Presidio) to locally replace names, social security numbers, and email addresses with placeholders like [PERSON] or [EMAIL] before logging. The assumption here is that PII detection models are about 95% accurate; manual review or strict retention remains necessary.
  3. Shadow Logs with Strict TTL: If you truly need the raw text for debugging purposes, store it in an isolated, encrypted database with highly restricted access rights (Role-Based Access Control) and an automatic Time-To-Live (TTL) of, for example, 3 or 7 days. After that period, the text is permanently deleted, while the metadata (tokens, latency) is preserved in your regular logs.

Tracing Across Multiple Steps (Agents & Chains)

Modern LLM applications rarely consist of a single simple call. Systems that use function calling or complex RAG architectures often execute a chain of actions. For example: a user asks a question, the system retrieves data from a vector database, calls an internal API via an LLM tool, and uses the results to generate a final answer.

If the response takes 15 seconds, where is the delay? To answer this, you borrow concepts from OpenTelemetry, specifically Traces and Spans.

By providing each log entry with a trace_id, a span_id, and a parent_span_id, you can generate a waterfall chart in tools like Jaeger or Datadog. This allows you to see at a glance that the slow response was not due to the LLM model, but to a timeout in Span 3 (the internal weather API).

A Concrete Log Schema (JSON)

To make the above theory concrete, below is an example of a robust JSON log schema that you should write to your backend per API call (i.e., per span).

{
  "timestamp": "2026-07-25T22:23:07Z",
  "environment": "production",
  "application_name": "customer_support_bot",
  
  "trace": {
    "trace_id": "5b8d9c2e-4a6f-11ef-8b2b-0242ac120002",
    "span_id": "a1b2c3d4-e5f6-7890",
    "parent_span_id": "c3d4e5f6-7890-a1b2"
  },
  
  "llm_config": {
    "provider": "openai",
    "model": "gpt-4-0613",
    "temperature": 0.2,
    "max_tokens": 1000
  },
  
  "content_metadata": {
    "prompt_hash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "tool_calls_requested": ["get_order_status"],
    "finish_reason": "stop"
  },
  
  "metrics": {
    "tokens_prompt": 450,
    "tokens_completion": 125,
    "tokens_total": 575,
    "latency_ttft_ms": 420,
    "latency_total_ms": 1850,
    "cost_estimated_usd": 0.023
  },
  
  "status": {
    "success": true,
    "http_code": 200,
    "provider_request_id": "req_8xV1..."
  }
}

This schema is completely flat and anonymized (no PII), but rich enough to build complex dashboards.

Troubleshooting: Finding Latency and Cost Spikes

With the data from the schema above, you are equipped to proactively find problems. Suppose the monthly bill from the LLM provider suddenly explodes. Through your dashboards, you can now group by prompt_hash or tool_calls_requested.

You might discover that a specific combination of a prompt and an internal database tool sends unnecessarily much context, causing tokens_prompt to structurally exceed 8000. Because you have the prompt_hash, you can search back in your version control to see which prompt template generates this hash, and optimize it.

For latency issues, you can aggregate by model and calculate the P95 (95th percentile) of latency_ttft_ms. If you see that gpt-4-turbo suddenly has a TTFT of 4000ms, whereas yesterday it was 400ms, you know immediately that the LLM provider is experiencing performance degradation, and you can automatically switch to a fallback model or another region.

Which Alerts Are Truly Useful?

Too many alerts lead to alert fatigue. Developers eventually ignore the notifications. Limit your LLM alerts to actionable signals:

  1. Error Rate (HTTP 4xx/5xx): Alert if more than 2% of requests fail within a 5-minute window. This usually indicates rate limit exhaustion or a provider outage.
  2. TTFT Degradation: Set an alert if the P90 of the Time To First Token rises above a critical threshold (e.g., > 2 seconds). This directly impacts the user experience and may require switching to a faster model.
  3. Cost or Token Limits (Budget Alerts): Set warnings as soon as the cumulative amount of cost_estimated_usd within one hour exceeds a certain threshold. This is your best defense against a DDoS attack or a runaway script that keeps making infinite calls.
  4. Frequency of 'finish_reason': 'length': If models frequently cut off due to reaching a token limit (instead of a natural 'stop'), it means the output for the user is incomplete. An increase in this metric requires immediate investigation into the max_tokens parameter used.

Conclusion

Observability for LLMs requires more than standard API monitoring. Due to the unpredictable nature of generative AI, you must meticulously record performance (especially tokens and first-byte latency), costs, and orchestration steps (traces). At the same time, ensure you respect privacy by hashing or strictly masking texts. With a well-thought-out log schema, you not only build a foundation for reliable AI applications, but also keep operational costs and user security tightly under control.