An observability dashboard for your LLM calls

Traditional Application Performance Monitoring (APM) fundamentally falls short as soon as a software architecture becomes dependent on external language models. Where traditional microservices fail with explicit HTTP status codes like 500 Internal Server Error or immediately visible spikes in server memory and CPU load, an LLM integration almost always fails silently. A provider's API endpoint happily returns a clean status code 200 OK, while response latency meanwhile creeps up to fifteen seconds, token consumption triples due to a prompt change, and the semantic quality of the generated text completely collapses.

Those who look exclusively at binary uptime or generic latency averages miss the crucial signals of operational degradation. This article falls within the quality assurance and change management pillar (A7); for the underlying data models and the capturing of spans and traces, we refer to the anchor article on observability and logging for LLM applications which covers the fundamentals of structured logging. In this guide, we expand those concepts into an operational observability dashboard that provides engineering teams with real-time insight into every individual interaction with a model provider.

The four pillars of LLM telemetry: TTTR

An effective dashboard for LLM calls structures its metrics around four specific axes: Traces, Tokens, Time, and Rates (summarized as TTTR). Unlike conventional APIs, these four variables are inextricably linked in AI models. After all, an increase in input tokens not only directly increases variable costs, but also extends the model's processing time and increases the likelihood of network timeouts.

A dashboard should never get stuck in one-way charts. The primary design goal is interactive trace correlation: when a team sees in a chart that P99 latency peaks at twenty seconds at 14:00, clicking that peak should immediately display the underlying traces, including the exact model version, token count, and any failed tool calls.

Span architecture and distributed tracing in multi-stage pipelines

In modern AI systems, an LLM call rarely stands alone. A user request often triggers a complex chain: first, vector embeddings are computed to retrieve documents, then the model generates a JSON structure for a database query, after which a second LLM call formulates the final response. Without distributed tracing, it is impossible to determine which link failed in the event of a slow response.

When an architecture uses a central proxy to distribute requests across multiple backends, the gateway injects tracing headers into every network step. Read the explanation of LLM API aggregators to see how a central routing layer helps standardize metadata and error handling across different providers. By traceparentpassing headers compliant with the W3C Trace Context standard, the coherence between frontend, application server, and model proxy is fully preserved.

{
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "parent_span_id": "5fb397be34d23b0f",
  "name": "llm.chat.completions",
  "start_time_unix_nano": 1786800000000000000,
  "end_time_unix_nano": 1786800001850200000,
  "attributes": {
    "gen_ai.system": "openai",
    "gen_ai.request.model": "gpt-4o",
    "gen_ai.response.model": "gpt-4o-2024-08-06",
    "gen_ai.usage.prompt_tokens": 1420,
    "gen_ai.usage.completion_tokens": 285,
    "gen_ai.usage.cached_tokens": 1024,
    "http.status_code": 200,
    "llm.ttft_ms": 420.5,
    "llm.total_duration_ms": 1850.2,
    "app.tenant_id": "tenant-corp-92",
    "app.environment": "production"
  }
}

The snippet above uses the OpenTelemetry Semantic Conventions for generative AI. By strictly adhering to these open standards instead of proprietary vendor formats, the dashboard can directly query uniform attributes such as gen_ai.usage.prompt_tokens, regardless of whether the underlying provider is OpenAI, Anthropic, or a locally running open-source model via vLLM.

Latency metrics dissected: TTFT, ITL, and queueing delays

Measuring solely the total processing time (end-to-end latency) provides a skewed picture of the user experience. With streaming responses via Server-Sent Events (SSE), a user already perceives an application as responsive as soon as the first words appear on the screen within half a second, even if the total text generation takes ten seconds. The dashboard should therefore break down the timing measurement into three components:

• Time To First Token (TTFT): The duration between sending the HTTP request and receiving the very first streaming chunk. This encompasses network latency, TLS handshake, any queue time at the provider, and the prefill computation over the entire prompt.
• Inter-Token Latency (ITL): The average time between consecutive tokens during the decoding phase. A stable ITL of 20 to 30 milliseconds per token ensures a smooth stream of text (33 to 50 tokens per second).
• Queueing Delay: The time a request spent waiting on the provider infrastructure before compute power on GPU clusters was allocated. Providers sometimes pass this along via response headers.

When TTFT suddenly doubles while the prompt token count remains unchanged, it almost always points to peak load or queuing at the provider. If ITL spikes suddenly, it indicates compute throttling. Displaying percentiles (P50, P90, P99) separately for both metrics makes infrastructural bottlenecks instantly traceable.

Monitoring token management, context growth, and caching ratios

Uncontrolled context window growth is one of the most common causes of soaring bills and sluggish applications. In chat interfaces or autonomous agent loops, previous interactions are often repeatedly appended without restrictions. As a result, the prompt grows exponentially with each turn of conversation until the context limit is hit or the cost per interaction becomes unsustainable.

The dashboard should break down token metrics into prompt tokens, generated completion tokens, and reused cache tokens. Major AI providers offer substantial discounts (often 50% to 80%) when static system prompts are retained in cluster memory. Visualizing the cache hit ratio in real time allows the engineering team to see immediately whether prompt refactoring is genuinely saving money.

Error analysis, application failure modes, and quality monitoring

When monitoring LLMs, the HTTP status code tells only a fraction of the story. Errors like HTTP 429 (Rate Limit Exceeded) or HTTP 503 (Service Unavailable) are easy to catch with alerts. Far more dangerous are application-level failures where the API returns an HTTP 200, but the payload is unusable. Examples include JSON schema violations, hallucinating non-existent facts, or model refusals caused by overzealous safety filters.

To detect semantic degradation, a dashboard combines quantitative status data with qualitative evaluation scores. For proven methods to systematically test generated answers for factual accuracy and source consistency, see the practical step-by-step guide on fact-checking AI responses on our guide platform. By asynchronously scoring periodic samples (such as 2% of all production responses) with a smaller evaluator model on metrics like faithfulness and schema compliance, a historical quality graph is established.

# Prometheus PromQL query voorbeelden voor LLM-monitoring

# 1. Percentage applicatieve fouten (HTTP fouten + ongeldige JSON payloads)
(
  sum(rate(llm_requests_total{status=~"error|invalid_schema"}[5m]))
  /
  sum(rate(llm_requests_total[5m]))
) * 100

# 2. P95 Time To First Token (TTFT) per model in seconden
histogram_quantile(0.95, sum(rate(llm_ttft_seconds_bucket[5m])) by (le, model))

# 3. Realtime geschatte kosten per minuut per tenant
sum(rate(llm_estimated_cost_usd_total[5m])) by (tenant_id) * 60

# 4. Prompt Caching efficiëntie (percentage tokens uit cache)
(
  sum(rate(llm_tokens_total{type="cached"}[5m]))
  /
  sum(rate(llm_tokens_total{type="prompt"}[5m]))
) * 100

Data sanitization, privacy, and PII masking in the pipeline

Logging entire user prompts and generated responses indiscriminately creates substantial legal and infrastructural risks. Prompts regularly contain personally identifiable information (PII), medical details, internal source code, or confidential financial figures. Storing these unfiltered in a central Elastic or ClickHouse cluster creates a vulnerable target for data breaches and violates GDPR principles regarding data minimization.

A secure observability pipeline implements data sanitization directly at the edge of the application (edge processing) before spans are forwarded to central storage. This also applies to the configuration of authentication keys: consult the guide on securely managing API keys to prevent secret tokens from accidentally ending up in trace attributes or error reports.

Therefore, a mature production environment employs a tiered data storage strategy:

• Metadata mode (100% of traffic): Tokens, response times, model names, tenant IDs, and calculated costs are permanently stored for dashboards and billing. The text fields (prompt and response) remain empty.
• Sampling for error analysis (100% of errors, 1% of success): Text payloads are only temporarily retained for failed requests or a very small random sample, with a strict automatic TTL (Time To Live) of, for example, 72 hours.
• PII masking via regex or local NER model: Before a span leaves the application server, sanitization filters replace email addresses, citizen service numbers (BSN), credit card details, and IP addresses with generic tokens such as [REDACTED_EMAIL].

Architecture for a self-hosted telemetry stack

Setting up an enterprise-grade observability dashboard does not require an expensive SaaS service. Combining open-source building blocks allows you to build a scalable stack that processes millions of spans per day without confidential data ever leaving your own network.

The architecture consists of four modular layers:

1. Instrumentation layer: The application code or API gateway uses the OpenTelemetry SDK to generate spans around each LLM call. Metadata such as tokens, latency, and tenant IDs are added as standard attributes.
2. OpenTelemetry Collector: A central collector receives traces via OTLP (gRPC or HTTP). The collector runs processors for batching, PII filtering, and sampling before forwarding the data.
3. Storage backends: A combination of ClickHouse for analytical traces and structured logs, and Prometheus (or Mimir) for high-speed time-series metrics and alerting. ClickHouse provides superior compression and blazing-fast querying capabilities across arbitrary JSON attributes.
4. Grafana visualization: The central dashboard where Grafana panels connect to ClickHouse and Prometheus. Here, real-time cost meters, latency heatmaps, and error overviews are clearly unified.

Concrete alerting thresholds and operational failure patterns

A dashboard is only valuable if the team is alerted in time before end users experience disruptions. Where traditional alerting often relies on static error rates, LLM monitoring requires dynamic thresholds that account for token volumes and financial budgets.

Four critical alert rules that should be configured directly on the dashboard and via notification systems:

• Budget burn-rate alert: Trigger a P1 alert as soon as estimated spend over a 1-hour rolling window exceeds 300% of the historical hourly average. This prevents an infinite agent loop from burning through thousands of euros in API credits in a short time.
• TTFT P95 Degradation: Raise an alarm when the 95th percentile of Time To First Token exceeds 3500 milliseconds for 5 consecutive minutes. This indicates provider-side overload and triggers an automatic failover to a secondary model.
• Structural Schema Validation Errors: Trigger a warning as soon as more than 2% of requests result in unparseable JSON. This points to altered instruction following after a provider update or a flawed prompt template.
• Token Context Spikes: Alert when average prompt token usage per user increases by more than 50% within fifteen minutes, indicating a failing context reduction algorithm.

Conclusion and operational assurance

Setting up a well-designed observability dashboard transforms LLM application management from blind faith into a manageable, measurable discipline. By systematically steering on Time To First Token, token efficiency, cache utilization, and application error rates, engineering teams regain control over external AI dependencies.

In a technological landscape where providers frequently update models, adjust compute quotas, and latency fluctuates, continuous telemetry is the only reliable compass. With standardized OpenTelemetry instrumentation, robust data sanitization, and real-time visualization, an AI integration remains reliable, fast, and within budget.