Learning Selective Merge Policies for Deadline-Constrained Coded Caching via Deep Reinforcement Learning

TL;DR

This paper introduces a deep reinforcement learning approach to optimize selective message merging in deadline-constrained coded caching, significantly reducing packet expiration and improving efficiency.

Contribution

It formulates the deadline-constrained coded delivery as a masked discrete-action queue control problem and trains a graph-attention policy network for optimal merging decisions.

Findings

Reduced broadcast-packet expiration ratio by 40.9% compared to baseline

Achieved best broadcast-efficiency score among coded multi-casting methods

Learned to merge only 31.8% of the time for tight deadline users

Abstract

With the coded caching, the server can use the information the users have cached to serve multiple users at a time by sending a single coded multi-casting message, i.e., the merged message, thereby relieving the peak network loads. However, for the delay-sensitive applications of the users, like the video streaming services, it becomes essential to choose which messages to merge online, considering the strict deadlines for each request. The problem, however, is that while the merge is helpful for the formation of the current coded multi-casting message, it can be harmful for the subsequent ones. We proposed a DRL-based solution that formulates the deadline-constrained coded delivery as a masked discrete-action queue-state control problem, while we trained a graph-attention policy network via proximal policy optimization. The policy network reduces the broadcast-packet expiration ratio $\rho$ by $40.9%$ ($0.208$ vs. $0.352$) with respect to the best coded multi-casting baseline (SACM++) on the uniform-demand benchmark, while also attaining the best broadcast-efficiency score $\sigma$ across the Track A battery among the coded multi-casting methods. The interesting fact we observed is that for the applications of the users with tight deadlines, the method of selective merging is better than the method of aggressive merging, i.e., the policy network learns to merge at only $\approx 31.8%$ rate, even though the same observation holds across the variations within the same simulator family.