# Reinforcement Learning for Adaptive Caching with Dynamic Storage Pricing

## TL;DR

This paper develops a reinforcement learning framework for adaptive caching in 5G small base stations, optimizing fetch-cache decisions under dynamic costs and storage constraints to improve network performance.

## Contribution

It introduces a novel RL-based approach for dynamic fetch-cache decision-making considering time-varying costs and capacity limitations in edge caching.

## Key findings

- RL-based algorithms outperform static caching policies.
- The approach effectively adapts to changing cost and demand patterns.
- Numerical results demonstrate improved network efficiency.

## Abstract

Small base stations (SBs) of fifth-generation (5G) cellular networks are envisioned to have storage devices to locally serve requests for reusable and popular contents by \emph{caching} them at the edge of the network, close to the end users. The ultimate goal is to shift part of the predictable load on the back-haul links, from on-peak to off-peak periods, contributing to a better overall network performance and service experience. To enable the SBs with efficient \textit{fetch-cache} decision-making schemes operating in dynamic settings, this paper introduces simple but flexible generic time-varying fetching and caching costs, which are then used to formulate a constrained minimization of the aggregate cost across files and time. Since caching decisions per time slot influence the content availability in future slots, the novel formulation for optimal fetch-cache decisions falls into the class of dynamic programming. Under this generic formulation, first by considering stationary distributions for the costs and file popularities, an efficient reinforcement learning-based solver known as value iteration algorithm can be used to solve the emerging optimization problem. Later, it is shown that practical limitations on cache capacity can be handled using a particular instance of the generic dynamic pricing formulation. Under this setting, to provide a light-weight online solver for the corresponding optimization, the well-known reinforcement learning algorithm, $Q$-learning, is employed to find optimal fetch-cache decisions. Numerical tests corroborating the merits of the proposed approach wrap up the paper.