Limited Associativity Makes Concurrent Software Caches a Breeze

TL;DR

This paper advocates for limited associativity in concurrent software caches, demonstrating it simplifies cache management, maintains high hit ratios, and significantly improves throughput in multi-threaded environments.

Contribution

It introduces limited associativity as a viable alternative to fully associative caches, enabling simpler and more efficient concurrent cache management schemes.

Findings

Limited associativity achieves similar hit ratios to fully associative caches.

Throughput improves by up to 5x in multi-threaded scenarios.

Simpler cache management schemes are possible with limited associativity.

Abstract

Software caches optimize the performance of diverse storage systems, databases and other software systems. Existing works on software caches automatically resort to fully associative cache designs. Our work shows that limited associativity caches are a promising direction for concurrent software caches. Specifically, we demonstrate that limited associativity enables simple yet efficient realizations of multiple cache management schemes that can be trivially parallelized. We show that the obtained hit ratio is usually similar to fully associative caches of the same management policy, but the throughput is improved by up to X5 compared to production-grade caching libraries, especially in multi-threaded executions.