Tight Rates for Bandit Control Beyond Quadratics

TL;DR

This paper introduces an algorithm that achieves near-optimal regret bounds for complex bandit control problems with adversarial and non-quadratic costs, surpassing previous bounds.

Contribution

It presents a novel algorithm that attains $ ilde{O}( oot{2} otimes T)$ regret for bandit control with non-quadratic costs, overcoming memory challenges and improving prior results.

Findings

Achieves $ ilde{O}( oot{2} otimes T)$ regret in bandit control.

Develops an improved bandit convex optimization algorithm with memory.

Demonstrates the effectiveness of reduction to memoryless BCO.

Abstract

Unlike classical control theory, such as Linear Quadratic Control (LQC), real-world control problems are highly complex. These problems often involve adversarial perturbations, bandit feedback models, and non-quadratic, adversarially chosen cost functions. A fundamental yet unresolved question is whether optimal regret can be achieved for these general control problems. The standard approach to addressing this problem involves a reduction to bandit convex optimization with memory. In the bandit setting, constructing a gradient estimator with low variance is challenging due to the memory structure and non-quadratic loss functions. In this paper, we provide an affirmative answer to this question. Our main contribution is an algorithm that achieves an $\tilde{O}(\sqrt{T})$ optimal regret for bandit non-stochastic control with strongly-convex and smooth cost functions in the presence of adversarial perturbations, improving the previously known $\tilde{O}(T^{2/3})$ regret bound from (Cassel and Koren, 2020. Our algorithm overcomes the memory issue by reducing the problem to Bandit Convex Optimization (BCO) without memory and addresses general strongly-convex costs using recent advancements in BCO from (Suggala et al., 2024). Along the way, we develop an improved algorithm for BCO with memory, which may be of independent interest.