[EAAPL-PLT006] LLM Caching Layer

Category: Platform Engineering Sub-category: Performance / Cost Optimisation Version: 1.1 Maturity: Proven Tags: caching, semantic-cache, exact-match-cache, cache-invalidation, prompt-cache, cost-reduction, latency, privacy Regulatory Relevance: Privacy Act, GDPR (cached PII), ISO 27001 (data retention)

1. Executive Summary

LLM inference is expensive and often redundant. Across enterprise deployments, 20–40% of LLM requests are near-duplicates of previous requests—particularly in customer service, FAQ answering, document summarisation, and search augmentation use cases. The LLM Caching Layer pattern systematically intercepts these redundant calls and serves responses from cache, reducing both token costs and end-user latency simultaneously.

This pattern defines two complementary caching mechanisms: exact-match caching for deterministic, templated prompts (100% identical prompt → 100% cache hit), and semantic caching for paraphrased or slightly varied prompts that should produce the same response (vector similarity above a configured threshold → cache hit). Together, these mechanisms create a caching layer that is transparent to consuming applications, requires no application code changes, and delivers measurable ROI from day one. The pattern also addresses the critical privacy and security concern of cached responses: ensuring cached content from one consumer is never served to another, and that PII-containing responses are handled with appropriate retention policies.

2. Problem Statement

Business Problem

LLM inference budgets are consumed by redundant computation. When a thousand customers ask the same product question in slightly different words, the enterprise pays for a thousand independent model inferences when a single cached response would suffice. This is not an engineering concern—it is a direct charge to the business unit's operational budget with no corresponding value.

Technical Problem

Model inference has latency that varies from 500ms to 15+ seconds depending on model and response length. For customer-facing features, this latency is a user experience problem. Without caching, every request waits for model inference even when the answer is known. Furthermore, streaming responses are difficult to cache, requiring specialised handling.

Symptoms

• Identical or near-identical prompts appearing in request logs at high frequency with no caching
• Customer-facing AI features with P95 latency >2 seconds for common queries
• LLM inference costs growing proportionally with user volume rather than with unique queries
• Teams separately implementing ad hoc caching solutions (in-memory, Redis) in their applications

Cost of Inaction

• 20–40% of LLM spend attributable to redundant inference with no caching in place
• Latency spikes during peak load as model providers throttle under increased concurrent requests
• Engineering effort duplicated as teams build bespoke caching independently

3. Context

When to Apply

• High volume of LLM requests with expected query repetition (FAQ, search, classification, summarisation)
• Latency improvement for AI features is a product quality objective
• LLM spend optimisation is active
• Centralised AI API Gateway (PLT002) exists as the appropriate host for shared caching

When NOT to Apply

• Highly personalised prompts where every request contains unique user context: cache hit rate will be negligible
• Creative generation use cases (story writing, code generation from unique specs): semantic similarity does not imply equivalent quality
• Prompts with real-time data requirements (current price, today's news): stale cached responses would be incorrect
• Applications requiring non-deterministic response variety: caching by definition reduces variety

Prerequisites

• AI API Gateway as host for the caching layer (PLT002)
• Redis or compatible vector-enabled cache infrastructure
• Embedding model (local or API) for semantic similarity computation
• Defined TTL policies per use case and corpus type
• Privacy assessment completed for cache content retention

Industry Applicability

4. Architecture Overview

The LLM Caching Layer is positioned as middleware within the AI API Gateway's request pipeline, executing after policy enforcement and before model routing. This positioning is deliberate: policy enforcement runs before caching to ensure that cache lookups do not bypass security controls, and caching runs before routing to prevent unnecessary routing computation for cache hits.

Exact-Match Caching targets the simplest case: when the complete prompt is byte-for-byte identical to a previous request. A SHA-256 hash of the canonical prompt (normalised for whitespace, with dynamic variables stripped) is used as the cache key. On a cache hit, the stored response is returned immediately with a X-Cache: HIT header. This mechanism is particularly effective for system-prompt-heavy applications where the system prompt constitutes >90% of the input tokens and changes rarely.

The cache key design for exact-match is subtle. The full prompt including the system prompt must be hashed, not just the user message. The model name and version must be included in the cache key, since the same prompt produces different outputs from different models. The response format parameters (temperature, top_p, max_tokens, output format) must also be included, since they affect the response. A cache key that ignores any of these dimensions risks serving a response produced under different conditions.

Semantic Caching handles the more complex case: prompts that are paraphrases of each other and should produce equivalent responses. The user-facing portion of the prompt (not the system prompt, which is typically stable) is embedded using a local embedding model into a dense vector. This vector is queried against a vector store (Redis with RediSearch, pgvector, or a dedicated vector database) using approximate nearest-neighbour search with cosine similarity. If the nearest stored prompt vector has a similarity score above the configured threshold, the associated stored response is returned.

The similarity threshold is the most operationally sensitive parameter in semantic caching. Too high (>0.99): essentially exact-match behaviour, low hit rate. Too low (<0.90): risk of serving an incorrect response to a superficially similar but semantically different prompt. The appropriate threshold depends on the use case: FAQ answering (high tolerance for paraphrase, threshold 0.92–0.95), code generation (low tolerance, threshold 0.98+), product description (medium tolerance, threshold 0.94–0.97). Thresholds should be validated against a test set of similar-but-different prompts to measure false positive rate.

Cache Invalidation is the hardest problem in caching, and LLM response caches are no exception. Three invalidation triggers are relevant: time-based TTL (responses expire after a configured period; appropriate for most use cases), corpus-update-triggered invalidation (when the knowledge base underlying the AI responses changes, all related cache entries must be invalidated; this requires tagging cache entries with corpus version metadata), and explicit invalidation (an API endpoint allowing platform operators or application owners to flush specific entries or entire namespaces). Corpus-update invalidation is the most complex; it requires the cache to maintain an index of which knowledge corpus version each entry was computed against.

Privacy and Tenancy Controls are non-negotiable. Cache entries must be scoped to the consumer namespace by default: a cached response for Team A's prompt must not be returned for Team B's identical prompt unless explicitly configured as a shared cache. For responses that may contain PII (personalised summaries, user-specific answers), caching must be disabled at the per-request level via a X-Cache-Control: no-store header or use-case configuration. All cache entries must respect their TTL strictly; there must be no mechanism to enumerate cached prompt content.

Cache Observability exposes the operational value of caching. Cache hit rate, cache size, eviction rate, average response latency (hit vs. miss), and attributable token savings are published as metrics. These metrics drive the FinOps dashboard's cache savings attribution and inform threshold tuning decisions.

5. Architecture Diagram

6. Components

7. Data Flow

Primary Flow — Semantic Cache Hit

Error Flow

8. Security Considerations

Tenant Isolation

• Cache namespace prefix is always scoped to consumer team namespace; {team}:{prompt_hash} prevents cross-tenant cache hits by default
• Shared cache pools (opt-in) require explicit platform team configuration and are only appropriate for non-personalised, non-confidential content (e.g., public FAQ)
• Redis ACLs enforce per-namespace read/write isolation; no consumer can access another tenant's cache entries

Privacy Controls

• Requests tagged with Cache-Control: no-store (from use-case configuration or explicit consumer header) bypass both exact and semantic cache
• Cache entries containing PII-derived responses must have reduced TTL (max 24 hours) and must be excluded from any cross-tenant sharing
• Cached responses are stored as ciphertext (AES-256) in Redis when classified above INTERNAL sensitivity level

Data Retention

• Cache TTLs are the mechanism for data retention compliance; they must be aligned with the privacy policy for the prompt content type
• Regular TTL audit to ensure cache retention does not exceed applicable data retention requirements

OWASP LLM Controls

9. Governance Considerations

Cache Policy Governance

• TTL policies per use case are defined by the Data Governance team and enforced by the platform; product teams cannot override TTLs beyond their configured maximum
• Semantic similarity thresholds per use case are reviewed by the prompt owner on a quarterly basis or after any quality incident attributable to cache false positives

Governance Artefacts

10. Operational Considerations

Monitoring

SLOs

Disaster Recovery

11. Cost Considerations

Cost Drivers

Optimisations

• Use a local embedding model (sentence-transformers, BGE) rather than an API embedding model to eliminate per-embedding costs and latency
• Configure Redis LRU eviction policy to automatically evict least-recently-used entries when memory pressure is high, maintaining cache quality without manual management
• Set TTLs aggressively short for use cases with dynamic context; longer for static knowledge-base use cases

Indicative Cost Range

12. Trade-Off Analysis

Caching Strategy Options

Similarity Threshold Options

Architectural Tensions

13. Failure Modes

Cascading Scenario

• Vector store failure + Redis failure simultaneously: Both caching layers unavailable; all requests fall through to model inference. At scale this can immediately exceed provider rate limits and trigger widespread 429 errors. Mitigation: circuit breaker on cache bypass path; pre-configured rate limit reduction to stay within provider rate limits during cache outage.

14. Regulatory Considerations

Privacy Act / GDPR

• Cached responses derived from personal information require the same legal basis as the original processing; caching must not extend the effective retention period beyond what the privacy policy allows
• TTL configuration is a data retention control; it must be managed by the Data Governance team, not left to engineering discretion
• Data subject deletion requests may require cache entries derived from that subject's data to be invalidated; the corpus-version invalidation mechanism supports this if prompts can be correlated to data subjects

ISO 27001

• Cache stores are information assets requiring appropriate access controls, encryption at rest, and audit logging in line with ISO 27001 Annex A.8

15. Reference Implementations

AWS

Azure

On-Premises

16. Related Patterns

17. Maturity Assessment

Overall Maturity: Proven LLM caching is production-proven at scale. Exact-match caching is commodity. Semantic caching via GPTCache and Redis Vector Search is operationally mature. Corpus-version invalidation is the least-standardised component.

Scoring Matrix

18. Revision History