Online Learning with Vector Costs and Bandits with Knapsacks

TL;DR

This paper introduces a new online learning framework with vector costs, providing algorithms that achieve near-optimal regret and competitive ratios in both stochastic and adversarial settings, with applications to Bandits with Knapsacks.

Contribution

It develops a reduction from multi-dimensional to single-dimensional online learning, enabling the use of classical algorithms for vector cost problems and improving results for Bandits with Knapsacks.

Findings

Achieves sublinear regret in stochastic settings.

Provides a tight $O( ext{min}igrace p, ext{log} digrace)$ competitive ratio for adversarial arrivals.

Improves the competitive ratio for adversarial Bandits with Knapsacks to $O( ext{log} d imes ext{log} T)$.

Abstract

We introduce online learning with vector costs (\OLVCp) where in each time step $t \in \{1,\ldots, T\}$, we need to play an action $i \in \{1,\ldots,n\}$ that incurs an unknown vector cost in $[0,1]^{d}$. The goal of the online algorithm is to minimize the $\ell_p$ norm of the sum of its cost vectors. This captures the classical online learning setting for $d=1$, and is interesting for general $d$ because of applications like online scheduling where we want to balance the load between different machines (dimensions). We study \OLVCp in both stochastic and adversarial arrival settings, and give a general procedure to reduce the problem from $d$ dimensions to a single dimension. This allows us to use classical online learning algorithms in both full and bandit feedback models to obtain (near) optimal results. In particular, we obtain a single algorithm (up to the choice of learning rate) that gives sublinear regret for stochastic arrivals and a tight $O(\min\{p, \log d\})$ competitive ratio for adversarial arrivals. The \OLVCp problem also occurs as a natural subproblem when trying to solve the popular Bandits with Knapsacks (\BwK) problem. This connection allows us to use our \OLVCp techniques to obtain (near) optimal results for \BwK in both stochastic and adversarial settings. In particular, we obtain a tight $O(\log d \cdot \log T)$ competitive ratio algorithm for adversarial \BwK, which improves over the $O(d \cdot \log T)$ competitive ratio algorithm of Immorlica et al. [FOCS'19].