RLCache: Automated Cache Management Using Reinforcement Learning

TL;DR

This paper presents RLCache, a reinforcement learning-based cache management system that improves cache hit rates and reduces storage needs by dynamically learning optimal caching strategies through multi-agent and multi-task architectures.

Contribution

It introduces a novel multi-task reinforcement learning framework for cache management, modeling cache decisions as a multi-decision control problem, and demonstrates superior performance over heuristic methods.

Findings

RL agents outperform heuristic algorithms in cache hit rate

Agents adapt to changing workloads dynamically

Framework scalable to different cache types

Abstract

This study investigates the use of reinforcement learning to guide a general purpose cache manager decisions. Cache managers directly impact the overall performance of computer systems. They govern decisions about which objects should be cached, the duration they should be cached for, and decides on which objects to evict from the cache if it is full. These three decisions impact both the cache hit rate and size of the storage that is needed to achieve that cache hit rate. An optimal cache manager will avoid unnecessary operations, maximise the cache hit rate which results in fewer round trips to a slower backend storage system, and minimise the size of storage needed to achieve a high hit-rate. This project investigates using reinforcement learning in cache management by designing three separate agents for each of the cache manager tasks. Furthermore, the project investigates two advanced reinforcement learning architectures for multi-decision problems: a single multi-task agent and a multi-agent. We also introduce a framework to simplify the modelling of computer systems problems as a reinforcement learning task. The framework abstracts delayed experiences observations and reward assignment in computer systems while providing a flexible way to scale to multiple agents. Simulation results based on an established database benchmark system show that reinforcement learning agents can achieve a higher cache hit rate over heuristic driven algorithms while minimising the needed space. They are also able to adapt to a changing workload and dynamically adjust their caching strategy accordingly. The proposed cache manager model is generic and applicable to other types of caches, such as file system caches. This project is the first, to our knowledge, to model cache manager decisions as a multi-task control problem.