Statistical Independence Aware Caching for LLM Workflows

TL;DR

This paper introduces Mnimi, a cache design pattern that ensures statistical independence in LLM workflows, improving reproducibility and efficiency without compromising probabilistic properties.

Contribution

The paper presents Mnimi, a novel cache design pattern that enforces statistical constraints in LLM workflows through type encapsulation, addressing limitations of existing caching systems.

Findings

Mnimi improves reproducibility and debugging in LLM workflows.

It reduces time and cost in LLM-based systems.

Mnimi maintains statistical correctness in caching.

Abstract

Large language models (LLMs) inference is both expensive and slow. Local caching of responses offers a practical solution to reduce the cost and latency of LLM queries. In research contexts, caching also enhances reproducibility and provides flexibility for experimentation. However, naive reuse of cached responses compromises statistical independence, a critical property for probabilistic workflows. In applications of LLM for code, it underpins performance metrics such as Pass@k and uncertainty estimation, as well as algorithms like program repair loops and retries. Existing LLM caching systems lack ways to enforce statistical independence constraints. To address this, we introduce Mnimi, a cache design pattern that supports modular LLM workflows while ensuring statistical integrity at the component level. Its core innovation lies in encapsulating statistical constraints within the type of LLM references, allowing users to manage and transform these types according to the scope and requirements of their algorithm. We implemented this design pattern in Python using a combination of decorators and iterators over infinite sequences. A case study on SpecFix, an recent automated program specification repair system, highlights how Mnimi improves reproducibility, ease of debugging, time and cost efficiency while preserving statistical correctness.