Can Increasing the Hit Ratio Hurt Cache Throughput? (Long Version)

TL;DR

This paper investigates how increasing cache hit ratios can paradoxically reduce system throughput, challenging common assumptions and highlighting implications for future high-performance systems.

Contribution

It demonstrates through modeling, measurement, and simulation that higher hit ratios may harm throughput, a counterintuitive insight with relevance for cache algorithm design.

Findings

Higher hit ratios can decrease throughput in many caching algorithms.

The phenomenon is more pronounced with faster disks and more CPU cores.

The study combines queueing theory, implementation, and simulation for validation.

Abstract

Software caches are an intrinsic component of almost every computer system. Consequently, caching algorithms, particularly eviction policies, are the topic of many papers. Almost all these prior papers evaluate the caching algorithm based on its hit ratio, namely the fraction of requests that are found in the cache, as opposed to disk. The hit ratio is viewed as a proxy for traditional performance metrics like system throughput or response time. Intuitively it makes sense that higher hit ratio should lead to higher throughput (and lower response time), since more requests are found in the cache (low access time) as opposed to the disk (high access time). This paper challenges this intuition. We show that increasing the hit ratio can actually hurt the throughput (and response time) for many caching algorithms. Our investigation follows a three-pronged approach involving (i) queueing modeling and analysis, (ii) implementation and measurement, and (iii) simulation to validate the accuracy of the queueing model. We also show that the phenomenon of throughput decreasing at higher hit ratios is likely to be more pronounced in future systems, where the trend is towards faster disks and higher numbers of cores per CPU.