Session management and context window compaction via the LLM API

Stateless API architectures form the foundation of modern machine learning pipelines, but interactive applications such as customer service assistants, coding tools, and data analysis environments require continuous conversational continuity across dozens of turns. Anyone who blindly resends the entire conversation history with every successive interaction quickly runs into exponentially rising costs, hard rate limits, and a measurable degradation in reasoning quality. This article falls under pillar A5 (Request patterns & data contract) and builds on the principles for retrieving reliable structured output to enforce predictable, compacted context structures.

We cover the architecture of client-side and gateway-side session management, and methods for cyclically shrinking message sequences (context compaction). This article does not cover prompt version control; see version control for prompts in a codebase for the controlled rollout of prompt templates. Anyone who wants to understand how unchecked context growth directly drives up operational spending will find in-depth analysis in the article on what a long context window really costs.

The failure mode of naive session management: context bloat and attention dilution

The simplest method for maintaining a dialogue via an LLM API is to store an array of messages locally and send it in full with every turn to the messagesparameter. In a production environment, this primitive pattern fails structurally on three separate dimensions:

First, it causes a quadratic increase in token consumption. In a conversation of twenty interactions, the operator pays not only for the latest response but, at interaction twenty, again for the input tokens of interactions one through nineteen. This leads to rapid exhaustion of configured usage limits, as described in the guide on rate limits, tokens, and cost management. The cumulative input cost of turn $N$ scales as $O(N^2)$, which in long sessions leads to budget overruns within a matter of hours.

Second, the phenomenon of attention dilution occurs, also known as 'lost-in-the-middle.' As the context window grows to tens of thousands of tokens, the transformer model's self-attention mechanism loses precision on instructions buried in the middle of the conversation history. The model starts ignoring earlier constraints, repeats assumptions that were already refuted, or hallucinates inconsistent parameters that the user had explicitly corrected earlier.

Third, a serious degradation of Time-To-First-Token (TTFT) occurs. The underlying model must prefill the entire prompt on every API call. Without thoughtful caching, the end user's wait time triples within ten conversation turns, directly harming the interactive user experience.

Architecture patterns for session management: Client, API, and Gateway

Management of conversation state can be placed at three different levels in the application landscape. Each choice comes with specific trade-offs regarding latency, security, and scalability:

Client-side state: The client application (for example, a browser SPA or mobile app) keeps the full conversation history in local memory or IndexedDB and sends the assembled array to the backend with every request. While this keeps the backend fully stateless, the pattern is extremely vulnerable to manipulation (prompt injection via tampered historical messages) and leads to unnecessarily heavy network payloads from mobile devices.

Application backend session storage: The backend stores the active message history in a fast key-value store (such as Redis or DynamoDB), linked to a cryptographically generated session ID. The frontend sends only the new user input and the session ID. The backend service loads the history, applies compaction, calls the LLM API, and saves the updated state. This is the most common industry standard for enterprise applications.

LLM Gateway managed context: A central API gateway intercepts traffic to external model providers, manages session state, and runs transparent compaction algorithms before the payload reaches the provider. This relieves application developers of repetitive context logic and centralizes monitoring and caching at a single infrastructure point.

Three compaction patterns compared

To keep an interactive session within tight budget limits without losing historical facts, three leading strategies exist. The matrix below lays out the architectural properties side by side:

For simple question-and-answer flows, a rolling window suffices, but for advanced business assistants, structured extraction is necessary. A broader conceptual comparison between loose prompts and advanced memory management can be found on the community platform via the guide to context window management.

Pattern 1: Rolling token-bounded FIFO buffer

The most basic compaction mechanism applies a hard token ceiling to the active conversation history. Rather than crudely trimming based on message count (which can vary widely in length), the algorithm trims based on precisely measured BPE tokens (Byte Pair Encoding). The initial system message always stays anchored at position zero, while the oldest user and assistant messages are removed in pairs once the threshold is exceeded.

The weakness of this pattern is evident: if a user states a crucial constraint in turn two (for example, "write only TypeScript and use no external dependencies"), this rule silently disappears from the window once turn twelve is reached. Use this approach only for short-cycle tasks where earlier interactions contain no binding constraints for later steps.

Pattern 2: Recursive semantic summarization

In recursive summarization, the application layer splits the context into three functional zones: the fixed system message, a compressed status block (the running summary of closed interactions), and an active working buffer with the most recent interactions (for example, the last 4 to 6 messages).

Once the total token size of the working buffer passes a predefined threshold percentage (the trigger threshold, for example 75% of the token budget), the orchestrator activates a compaction task. This merges the existing status summary and the oldest messages from the working buffer into a new consolidated text block. This significantly reduces token size, but introduces the risk of semantic drift: after four consecutive summarization cycles, specific technical details, such as error codes, file names, or exact numeric variables, can gradually blur or mutate.

Pattern 3: Deterministic extraction of entities and state

To effectively eliminate semantic drift, this pattern isolates the conversation context in a strict JSON schema. Instead of a free-text summary, a background process generates a state object that reflects the current state of the session (for example, selected filters, account IDs, established software requirements, and still-open questions).

By enforcing the state object through structured output, the context remains deterministic and programmatically manipulable by backend code. In the prompt injection, this state object is presented as JSON in the developer message. This means the raw chat history doesn't need to be kept in the context window to guarantee factual precision over long sessions.

Implementation: A compaction layer with sliding window and status block

The Python example below shows a production-ready session manager. The implementation combines a fixed system message, a compact status summary, and a dynamic sliding window that accounts for token limits and robust error handling during network failures.

import os
import time
import tiktoken
import httpx

class SessionContextManager:
  def __init__(self, system_instruction: str, max_context_tokens: int = 4000):
    self.system_instruction = system_instruction
    self.max_context_tokens = max_context_tokens
    self.summary_state = ""
    self.history = []
    self.tokenizer = tiktoken.get_encoding("cl100k_base")

  def _count_tokens(self, text: str) -> int:
    return len(self.tokenizer.encode(text))

  def add_message(self, role: str, content: str):
    self.history.append({"role": role, "content": content})

  def _build_payload_messages(self) -> list:
    messages = [{"role": "system", "content": self.system_instruction}]
    if self.summary_state:
      messages.append({
        "role": "system", 
        "content": f"Status van eerdere interacties:\n{self.summary_state}"
      })
    messages.extend(self.history)
    return messages

  def compact_if_needed(self, client: httpx.Client, api_url: str, api_key: str):
    total_tokens = sum(self._count_tokens(m["content"]) for m in self._build_payload_messages())
    
    # Activeer compaction zodra 80% van het tokenbudget is bereikt
    if total_tokens < (self.max_context_tokens * 0.8) or len(self.history) <= 4:
      return

    # Pak de oudere helft van de actieve historie om samen te vatten
    split_point = len(self.history) - 4
    to_compact = self.history[:split_point]
    self.history = self.history[split_point:]

    compact_prompt = (
      "Vat de volgende interacties feitelijk en compact samen in maximaal 100 woorden. "
      "Behoud specifieke entiteiten, afspraken, foutcodes en variabelen:\n" +
      "\n".join([f"{m['role']}: {m['content']}" for m in to_compact])
    )

    payload = {
      "model": "gpt-4o-mini",
      "messages": [
        {"role": "system", "content": "Je bent een context-compressor. Reageer strikt zakelijk."},
        {"role": "user", "content": compact_prompt}
      ],
      "temperature": 0.0
    }

    try:
      response = client.post(
        api_url,
        json=payload,
        headers={"Authorization": f"Bearer {api_key}"},
        timeout=10.0
      )
      response.raise_for_status()
      new_summary = response.json()["choices"][0]["message"]["content"]
      self.summary_state = f"{self.summary_state}\n{new_summary}".strip()
    except httpx.HTTPError as err:
      # Foutmitigatie: herstel de historie indien samenvatting mislukt
      self.history = to_compact + self.history
      raise RuntimeError(f"Context compaction gefaald: {err}")

  def prepare_request(self, client: httpx.Client, api_url: str, api_key: str) -> list:
    self.compact_if_needed(client, api_url, api_key)
    return self._build_payload_messages()

Interaction with KV cache and context caching

Context compaction must not be designed in isolation from modern provider architectures. Major LLM providers apply automated prefix caching (KV caching). Here, the provider recognizes identical token sequences at the start of the prompt and reuses the already-computed key-value attention vectors. This leads to discounts of 50% or more on input tokens and a drastically lower TTFT.

Carelessly designed session management constantly breaks this cache consistency. If a session manager places a dynamic timestamp, a current clock time, or a continuously changing summary right at the top of the prompt on every turn, prefix matching fails at the provider. To make optimal use of caching of LLM responses and prefixes, the payload should be structured hierarchically:

First, place the fully static system prompt and fixed tool schemas at the front of the context window. Second, place the compacted status block (which changes only periodically and incrementally) right after that. Place the volatile, recent chat history as the last element. This keeps the largest and heaviest part of the context static and suitable for the gateway's or provider's prefix cache.

Measurement methods for compaction quality and context retention

To evaluate whether a compaction algorithm functions successfully in production, purely intuitive spot checks are not enough. We use three quantitative metrics to monitor session quality:

1. Compression Ratio (CR): The ratio between the token count of the raw conversation history and the compacted representation. Formula: $\text{CR} = 1 - (\text{Tokens}_{\text{compacted}} / \text{Tokens}_{\text{raw}})$. A healthy pipeline achieves a CR between 0.65 and 0.85 without significant precision loss.

2. Entity Recall Accuracy (ERA): An automated test procedure in which an evaluation model checks whether predefined facts (such as product numbers, dates, or user restrictions) mentioned in early turns can still be correctly reproduced from the compacted state after multiple compaction passes.

3. Compaction Latency Overhead (CLO): The p95 latency of the compaction step itself. If executed inline, this measures the extra delay the end user experiences during a compaction turn compared to a regular interaction turn.

Edge cases and complex scenarios in production

In complex production environments, edge cases often arise that derail a simple FIFO or summarization loop:

Multimodal sessions with images and documents

When a user uploads an image or a large PDF document, this interaction consumes thousands of tokens. A classic text summarizer cannot easily compress binary or multimodal tokens. The robust solution is to store a textual description of the visual input in the status block immediately after processing, and to remove the heavy multimodal data from the active history after one turn.

Large tool outputs and JSON results

When an LLM calls an external tool (for example, a database query that returns 50 rows), this blows up the history in one go. The application must aggressively filter tool outputs or convert them to a compact schema before adding them to the session history. Never send raw SQL dumps or complete API responses back into the chat history.

User corrections and contradictions

If a user says: "Forget what I just said about budget X, the budget is now Y," a naive summarizer might retain both numbers ("originally X, later changed to Y"). This creates confusion in later reasoning steps. Deterministic state extraction via a JSON schema simply overwrites the field budget, eliminating contradictions for good.

Error paths and recovery strategies

When running automated context compaction, specific error conditions can occur that threaten the robustness of the application:

1. Compaction timeout during an active user session

When the underlying API call for the summary is delayed or fails with an HTTP 504 gateway timeout, the active user session must not be blocked. The session manager should degrade gracefully: perform a temporary FIFO truncation on the oldest messages, or send the uncompacted payload once, provided the absolute model maximum allows it.

2. Hallucination in the summary

The compression model can distort facts (for example, "the user wants to cancel the subscription" instead of "the user is asking about the cancellation notice period"). Mitigate this by running compaction prompts at a low temperature (0.0), providing specific extraction guidelines, and always verifying critical state parameters through deterministic validation rules.

3. Token budget overrun due to tool loops

When an agent gets stuck in a repeating loop of tool calls, the context window can fill up within seconds. The session manager must enforce a hard limit on the number of consecutive tool interactions within a single turn and intervene once a preset token limit is approached.

Cost, latency, and complexity: the operational trade-off

Context compaction is not a free operation. Anyone designing compaction must carefully weigh three trade-offs against each other:

Latency overhead: An inline summarization call adds between 400ms and 1500ms to the response time of the specific turn in which the compaction takes place. This can be avoided by running compaction asynchronously in a background task right after the previous response has been streamed to the user.

Financial cost: Generating summaries consumes output tokens, which are more expensive per token than input tokens. In short sessions (fewer than 5 interactions), context compaction is often more expensive than simply resending the full history. Apply compaction only once conversations structurally exceed the model's economic break-even point.

Operational complexity: Once state is extracted and stored, the system requires a redundant storage layer (such as Redis or PostgreSQL) to synchronize session state between requests. This introduces stateful components into a previously stateless microservice landscape.

Conclusion

Effective session management with LLM APIs requires a strict separation between static system instructions, a periodically consolidated status block, and a dynamic, recent working buffer. By combining targeted compaction algorithms with deterministic structured output and accounting for provider-side prefix caching, interactive LLM integrations remain scalable, predictable in cost, and consistent in response quality under heavy production load.