Caching LLM Responses: Faster and Cheaper

Why Caching in LLM Infrastructure?

Large Language Models (LLMs) are computationally intensive and inherently come with high Time-to-First-Token (TTFT) and substantial API costs per thousand tokens. In enterprise environments or high-traffic applications, repeatedly processing identical or highly similar user queries results in unnecessary waste of compute and capital.

By adding an intelligent caching layer in front of the LLM aggregation proxy, we intercept redundant requests immediately. This provides two direct benefits:

• Extreme latency reduction: From waiting seconds for inference to milliseconds for direct cache hits.
• Cost savings: No token costs for repeated queries, protecting the operational margin of your SaaS application.

Prompt Caching vs. Response Caching

Within LLM architectures, we distinguish between two fundamental caching mechanisms that complement each other perfectly:

Architecture & Pseudocode

A robust API gateway implements a layered strategy. First, it checks if there is an exact match in the response cache. If not, it checks whether prompt caching can be applied to the context before calling the upstream LLM.

// Pseudocode for API Gateway Cache Flow
function handleLLMRequest(incomingRequest) {
// Step 1: Generate deterministic hash of prompt and model configuration
const cacheKey = generateSHA256(incomingRequest.prompt + incomingRequest.model);

// Step 2: Check Response Cache (e.g., Redis)
const cachedResponse = RedisCache.get(cacheKey);
if (cachedResponse && !isExpired(cachedResponse)) {
return {
source: "response_cache",
latency_ms: 12,
data: cachedResponse.output
};
}

// Step 3: Check if Prompt Caching is possible (KV cache at provider)
let optimizedPayload = incomingRequest.payload;
if (hasStaticSystemPrompt(incomingRequest)) {
optimizedPayload = attachPromptCacheHeaders(incomingRequest);
}

// Step 4: Upstream LLM Call (Aggregator)
const startTime = getCurrentTimestamp();
const llmResponse = UpstreamLLM.complete(optimizedPayload);
const latency = getCurrentTimestamp() - startTime;

// Step 5: Store result in cache for future queries
RedisCache.set(cacheKey, {
output: llmResponse.text,
timestamp: getCurrentTimestamp()
}, TTL_SECONDS);

return {
source: "llm_upstream",
latency_ms: latency,
data: llmResponse.text
};
}

Cache Invalidation and Semantic Caching

The biggest danger of caching with LLMs is outdated or incorrect information. Because users rarely use the exact same phrasing, traditional string matching often falls short.

• TTL (Time-To-Live): Set strict expiration times for dynamic topics (e.g., 1 hour for news-related queries, 7 days for static documentation).
• Semantic Caching: Instead of exact string matching, use embeddings and vector similarity (cosine distance > 0.95) to recognize that a new query is effectively identical to a previous one.
• Event-driven invalidation: Automatically clear cache records as soon as underlying sources (such as RAG vector databases or product catalogs) are updated.

When to Cache and When Not to?

Not every LLM interaction is suitable for caching. A comparison of use cases: