Piecewise-Stationary Multi-Objective Multi-Armed Bandit with Application to Joint Communications and Sensing

TL;DR

This paper introduces a novel Pareto UCB-based algorithm with change detection for multi-objective multi-armed bandits in dynamic environments, achieving near-optimal regret bounds and demonstrated effectiveness in communication and sensing applications.

Contribution

It develops a new algorithm for multi-objective bandits with piecewise-stationary rewards, providing theoretical regret bounds and practical validation in communication systems.

Findings

Regret bound of order γ_T log(T/γ_T) with known breakpoints

Regret bound of order γ_T log(T) without known breakpoints

Demonstrated efficiency in synthetic, real-world, and communication system datasets

Abstract

We study a multi-objective multi-armed bandit problem in a dynamic environment. The problem portrays a decision-maker that sequentially selects an arm from a given set. If selected, each action produces a reward vector, where every element follows a piecewise-stationary Bernoulli distribution. The agent aims at choosing an arm among the Pareto optimal set of arms to minimize its regret. We propose a Pareto generic upper confidence bound (UCB)-based algorithm with change detection to solve this problem. By developing the essential inequalities for multi-dimensional spaces, we establish that our proposal guarantees a regret bound in the order of $\gamma_T\log(T/{\gamma_T})$ when the number of breakpoints $\gamma_T$ is known. Without this assumption, the regret bound of our algorithm is $\gamma_T\log(T)$. Finally, we formulate an energy-efficient waveform design problem in an integrated communication and sensing system as a toy example. Numerical experiments on the toy example and synthetic and real-world datasets demonstrate the efficiency of our policy compared to the current methods.