Non-stationary Bandit Convex Optimization: A Comprehensive Study

TL;DR

This paper studies non-stationary bandit convex optimization, proposing algorithms that adapt to environment changes and achieve minimax-optimal regret bounds under various measures of non-stationarity.

Contribution

It introduces TEWA-SE, a polynomial-time algorithm for strongly convex losses, and cExO, a non-polynomial method for general convex losses, both achieving optimal regret bounds in non-stationary settings.

Findings

TEWA-SE is minimax-optimal for strongly convex losses with known non-stationarity measures.

cExO achieves minimax-optimal regret for general convex losses, improving bounds related to path-length.

Algorithms adapt to unknown non-stationarity measures using the Bandit-over-Bandit framework.

Abstract

Bandit Convex Optimization is a fundamental class of sequential decision-making problems, where the learner selects actions from a continuous domain and observes a loss (but not its gradient) at only one point per round. We study this problem in non-stationary environments, and aim to minimize the regret under three standard measures of non-stationarity: the number of switches $S$ in the comparator sequence, the total variation $\Delta$ of the loss functions, and the path-length $P$ of the comparator sequence. We propose a polynomial-time algorithm, Tilted Exponentially Weighted Average with Sleeping Experts (TEWA-SE), which adapts the sleeping experts framework from online convex optimization to the bandit setting. For strongly convex losses, we prove that TEWA-SE is minimax-optimal with respect to known $S$ and $\Delta$ by establishing matching upper and lower bounds. By equipping TEWA-SE with the Bandit-over-Bandit framework, we extend our analysis to environments with unknown non-stationarity measures. For general convex losses, we introduce a second algorithm, clipped Exploration by Optimization (cExO), based on exponential weights over a discretized action space. While not polynomial-time computable, this method achieves minimax-optimal regret with respect to known $S$ and $\Delta$, and improves on the best existing bounds with respect to $P$.