Rate-Constrained Remote Contextual Bandits

TL;DR

This paper investigates a rate-constrained multi-agent contextual bandit problem, analyzing the fundamental limits of policy compression and proposing practical coding schemes to optimize regret under communication constraints.

Contribution

It introduces a theoretical framework for rate-limited policy transmission in CMABs, characterizes the information-theoretic limits, and proposes practical compression schemes.

Findings

Identifies two rate regions with linear and sub-linear regret.

Derives the fundamental limits of lossy compression for the policy.

Proposes a practical coding scheme with numerical validation.

Abstract

We consider a rate-constrained contextual multi-armed bandit (RC-CMAB) problem, in which a group of agents are solving the same contextual multi-armed bandit (CMAB) problem. However, the contexts are observed by a remotely connected entity, i.e., the decision-maker, that updates the policy to maximize the returned rewards, and communicates the arms to be sampled by the agents to a controller over a rate-limited communications channel. This framework can be applied to personalized ad placement, whenever the content owner observes the website visitors, and hence has the context, but needs to transmit the ads to be shown to a controller that is in charge of placing the marketing content. Consequently, the rate-constrained CMAB (RC-CMAB) problem requires the study of lossy compression schemes for the policy to be employed whenever the constraint on the channel rate does not allow the uncompressed transmission of the decision-maker's intentions. We characterize the fundamental information theoretic limits of this problem by letting the number of agents go to infinity, and study the regret that can be achieved, identifying the two distinct rate regions leading to linear and sub-linear regrets respectively. We then analyze the optimal compression scheme achievable in the limit with infinite agents, when using the forward and reverse KL divergence as distortion metric. Based on this, we also propose a practical coding scheme, and provide numerical results.