Modeling and Optimizing Latency for Delayed Hit Caching with Stochastic Miss Latency

TL;DR

This paper analyzes the impact of stochastic fetch latencies on delayed hit caching and introduces a variance-aware caching strategy that improves latency performance.

Contribution

It provides the first theoretical analysis of delayed hits with stochastic latencies and proposes a novel cache eviction method based on variance considerations.

Findings

Significant latency reduction on synthetic datasets (3%-30%)

Moderate latency reduction on real-world traces (1%-7%)

First analytical expressions for mean and variance of delays with stochastic fetches

Abstract

Caching is crucial for system performance, but the delayed hit phenomenon, where requests queue during lengthy fetches after a cache miss, significantly degrades user-perceived latency in modern high-throughput systems. While prior works address delayed hits by estimating aggregate delay, they universally assume deterministic fetch latencies. This paper tackles the more realistic, yet unexplored, scenario where fetch latencies are stochastic. We present, to our knowledge, the first theoretical analysis of delayed hits under this condition, deriving analytical expressions for both the mean and variance of the aggregate delay assuming exponentially distributed fetch latency. Leveraging these insights, we develop a novel variance-aware ranking function tailored for this stochastic setting to guide cache eviction decisions more effectively. The simulations on synthetic and real-world datasets demonstrate that our proposed algorithm significantly reduces overall latency compared to state-of-the-art delayed-hit strategies, achieving a $3\%-30\%$ reduction on synthetic datasets and approximately $1\%-7\%$ reduction on real-world traces.