A Learned Cache Eviction Framework with Minimal Overhead

TL;DR

This paper presents MAT, a framework that integrates machine learning with traditional cache algorithms to significantly reduce prediction overhead while maintaining cache efficiency across various workloads.

Contribution

MAT introduces a novel approach that uses heuristic filters to minimize ML predictions in cache eviction, improving practicality for high-throughput systems.

Findings

Reduces ML predictions per eviction from 63 to 2

Achieves comparable cache miss ratios to state-of-the-art ML caches

Maintains request rates similar to traditional LRU caches

Abstract

Recent work shows the effectiveness of Machine Learning (ML) to reduce cache miss ratios by making better eviction decisions than heuristics. However, state-of-the-art ML caches require many predictions to make an eviction decision, making them impractical for high-throughput caching systems. This paper introduces Machine learning At the Tail (MAT), a framework to build efficient ML-based caching systems by integrating an ML module with a traditional cache system based on a heuristic algorithm. MAT treats the heuristic algorithm as a filter to receive high-quality samples to train an ML model and likely candidate objects for evictions. We evaluate MAT on 8 production workloads, spanning storage, in-memory caching, and CDNs. The simulation experiments show MAT reduces the number of costly ML predictions-per-eviction from 63 to 2, while achieving comparable miss ratios to the state-of-the-art ML cache system. We compare a MAT prototype system with an LRU-based caching system in the same setting and show that they achieve similar request rates.