Optimal and Scalable Caching for 5G Using Reinforcement Learning of Space-time Popularities

TL;DR

This paper proposes a reinforcement learning-based caching strategy for 5G small base stations that adaptively learns space-time content popularities, optimizing cache performance amid dynamic user demands and limited resources.

Contribution

It introduces a scalable, online Q-learning framework with function approximation for optimal cache management considering complex popularity dynamics.

Findings

The RL approach outperforms traditional caching methods in dynamic scenarios.

Linear function approximation improves convergence speed and reduces complexity.

Numerical results validate the effectiveness of the proposed caching strategy.

Abstract

Small basestations (SBs) equipped with caching units have potential to handle the unprecedented demand growth in heterogeneous networks. Through low-rate, backhaul connections with the backbone, SBs can prefetch popular files during off-peak traffic hours, and service them to the edge at peak periods. To intelligently prefetch, each SB must learn what and when to cache, while taking into account SB memory limitations, the massive number of available contents, the unknown popularity profiles, as well as the space-time popularity dynamics of user file requests. In this work, local and global Markov processes model user requests, and a reinforcement learning (RL) framework is put forth for finding the optimal caching policy when the transition probabilities involved are unknown. Joint consideration of global and local popularity demands along with cache-refreshing costs allow for a simple, yet practical asynchronous caching approach. The novel RL-based caching relies on a Q-learning algorithm to implement the optimal policy in an online fashion, thus enabling the cache control unit at the SB to learn, track, and possibly adapt to the underlying dynamics. To endow the algorithm with scalability, a linear function approximation of the proposed Q-learning scheme is introduced, offering faster convergence as well as reduced complexity and memory requirements. Numerical tests corroborate the merits of the proposed approach in various realistic settings.