AI Systems
Caching — Reducing Latency and Cost
LLM inference is expensive in both latency and dollars per token. Caching stores and reuses previously computed results so repeated or near-duplicate queries never reach the model at all. A well-tuned caching layer routinely cuts inference spend by 30 to 80 percent and drops p50 latency from seconds to tens of milliseconds on a hit — this is the pattern interviewers and reviewers read as evidence someone thinks about unit economics, not only accuracy.
- What it is
- Storing and reusing previously computed model outputs so repeated or near-duplicate requests skip inference entirely.
- Mental model
- Don't ask the expert the same question twice — write the answer down the first time and check your notes before asking again.
- Key numbers
- A well-tuned cache cuts inference spend 30–80%; semantic cache similarity thresholds typically sit 0.90–0.97 depending on false-positive tolerance.
- Reach for it when
- Query volume is high and a meaningful fraction of traffic is exact or near-duplicate — the common case for FAQ-style and support traffic.
Exact-match caching
Hash the normalized prompt (whitespace-trimmed, case-folded where safe) and use it as a key in Redis or Memcached. Seeing the same key again returns the cached response without ever calling the model. Hit rates are low for free-form conversational text where phrasing varies, but high for automated pipelines and batch jobs with canonical inputs. It's trivial to implement, carries zero risk of returning a semantically wrong answer, and should be the first layer in any caching stack because it's nearly free.
Semantic caching
Embed incoming queries and search the cache by cosine similarity rather than exact string match — if a new query's embedding falls within a similarity threshold of a cached query (typically 0.90 to 0.97, depending on tolerance for false positives), the cached response comes back. This lifts hit rates substantially for conversational traffic where users ask the same question differently worded. The threshold is the entire design problem: too low and a wrong cached answer gets served to a different question, too high and hit rate collapses back toward exact-match levels. Tune it empirically against a labelled set of near-duplicate and non-duplicate pairs, and treat it as a metric monitored continuously, not set once at launch.
Prefix and KV caching
This operates at the model-serving layer, not the application layer. When multiple requests share a common prompt prefix — a long system prompt, few-shot exemplars, a large retrieved context reused across turns — the transformer's key-value attention states for that shared prefix are computed once and reused, avoiding redundant forward passes. Most major inference providers expose this automatically or via an explicit cache-control parameter, and savings on time-to-first-token can exceed 80 to 90 percent for long shared prefixes. Understanding this distinguishes candidates who think about model-serving internals from those who only think in application-layer terms.
Productionizing this pattern
- Cache invalidation is the hard problem, not the lookup. If the underlying knowledge base or RAG index changes, stale entries silently serve outdated answers — tie TTL to content update cadence, or invalidate keyed on a content-version hash rather than time alone.
- Layer all three types rather than choosing one: exact-match first (near-zero cost), semantic cache second (paraphrase coverage), prefix caching underneath both for anything that still reaches the model.
- Monitor cache hit rate as a first-class production metric alongside cost per request — a sudden drop is often the earliest signal of a shift in user query distribution.
- Caching is unsafe for personalized or time-sensitive responses (account balances, live sensor readings) unless the cache key includes the relevant personalization or timestamp dimension.
Real-world applications
- FAQ and support systems — a large fraction of traffic is variations of a small set of common questions — the textbook semantic-caching win.
- Batch/automated pipelines — canonical, repeated inputs make exact-match caching alone highly effective.
- Multi-turn chat with a long system prompt — prefix/KV caching makes the fixed system prompt essentially free across turns.
- Cost-sensitive high-volume products — anywhere the same handful of questions dominate traffic, caching is usually the single highest-leverage cost lever before touching the model itself.
Embedded applications
- Local-first response caching — a battery-powered or connectivity-limited device (a voice-controlled appliance, a field sensor gateway) benefits disproportionately from exact-match caching, since every cache hit avoids not just cost but a radio wake-up and a round trip over a metered or intermittent link — the same "cheap check before the expensive operation" instinct as clock stretching avoiding wasted bus cycles.
- KV-cache reuse on-device — when a small model runs directly on an NPU-equipped board, prefix/KV caching is the difference between a snappy repeated interaction and re-running the full forward pass every time — directly analogous to why a system prompt should stay fixed and short on constrained hardware.
- Cache as a bandwidth-cost tradeoff — for a fleet of devices talking to a cloud LLM over MQTT or cellular, a local semantic cache trades a small amount of on-device compute and flash for a large reduction in metered data usage — the embedded equivalent of the FAQ interview question's "60% of queries are duplicates" scenario, just with dollars-per-token replaced by dollars-per-megabyte.
- Staleness matters more, not less — a cached "is this valid" answer about live sensor data or actuator state is actively dangerous on embedded systems in a way it usually isn't for a chatbot — the personalized/time-sensitive caution above is not optional here, it's a safety requirement tied directly into Guardrails & Defensive UX.
Gotchas
- A semantic cache threshold tuned once at launch and never revisited drifts out of calibration as query distribution shifts — treat it as a live metric, not a constant.
- Caching personalized or live data without including that dimension in the cache key is a classic, easy-to-miss production bug — it looks correct in every test that happens to use the same account or the same moment in time.
- A cache hit rate that suddenly drops is a leading indicator worth alerting on — it usually means either a content change invalidated a large slice of the cache, or user query patterns genuinely shifted.