Asynchronous Verified Semantic Caching for Tiered LLM Architectures

TL;DR

Krites is an asynchronous caching policy for tiered LLM architectures that uses LLM judgments to expand static cache coverage without increasing latency, significantly reducing inference costs.

Contribution

It introduces an LLM-judged asynchronous verification mechanism that enhances static cache effectiveness in tiered LLM systems without affecting response latency.

Findings

Increases static cache hit rate by up to 3.9 times in simulations.

Maintains unchanged critical path latency.

Improves reuse of static responses in conversational and search workloads.

Abstract

Large language models (LLMs) now sit in the critical path of search, assistance, and agentic workflows, making semantic caching essential for reducing inference cost and latency. Production deployments typically use a tiered static-dynamic design: a static cache of curated, offline vetted responses mined from logs, backed by a dynamic cache populated online. In practice, both tiers are commonly governed by a single embedding similarity threshold, which induces a hard tradeoff: conservative thresholds miss safe reuse opportunities, while aggressive thresholds risk serving semantically incorrect responses. We introduce Krites, an asynchronous, LLM-judged caching policy that expands static coverage without changing serving decisions. On the critical path, Krites behaves exactly like a standard static threshold policy. When the nearest static neighbor of the prompt falls just below the static threshold, Krites asynchronously invokes an LLM judge to verify whether the static response is acceptable for the new prompt. Approved matches are promoted into the dynamic cache, allowing future repeats and paraphrases to reuse curated static answers and expanding static reach over time. In trace-driven simulations on conversational and search workloads, Krites increases the fraction of requests served with curated static answers (direct static hits plus verified promotions) by up to3.9 times for conversational traffic and search-style queries relative to tuned baselines, with unchanged critical path latency.