Alternating Regret for Online Convex Optimization

TL;DR

This paper demonstrates that the continuous Hedge algorithm achieves sublinear alternating regret in online convex optimization, enabling equilibrium convergence in two-player games, and introduces improved algorithms with better regret bounds for smooth losses.

Contribution

It extends alternating regret bounds from linear to general convex problems and proposes algorithms with improved regret rates for smooth and self-concordant losses.

Findings

Continuous Hedge achieves $ ilde{O}(d^{2/3}T^{1/3})$ regret in OCO.

Algorithms find Nash or correlated equilibria at $ ilde{O}(d^{2/3}/T^{2/3})$ rate.

New algorithms attain $ ilde{O}(T^{2/5})$ regret without dimension dependence.

Abstract

Motivated by alternating learning dynamics in two-player games, a recent work by Cevher et al.(2024) shows that $o(\sqrt{T})$ alternating regret is possible for any $T$-round adversarial Online Linear Optimization (OLO) problem, and left as an open question whether the same is true for general Online Convex Optimization (OCO). We answer this question in the affirmative by showing that the continuous Hedge algorithm achieves $\tilde{\mathcal{O}}(d^{\frac{2}{3}}T^{\frac{1}{3}})$ alternating regret for any adversarial $d$-dimensional OCO problems. We show that this implies an alternating learning dynamic that finds a Nash equilibrium for any convex-concave zero-sum games or a coarse correlated equilibrium for any convex two-player general-sum games at a rate of $\tilde{\mathcal{O}}(d^{\frac{2}{3}}/T^{\frac{2}{3}})$. To further improve the time complexity and/or the dimension dependence, we propose another simple algorithm, Follow-the-Regularized-Leader with a regularizer whose convex conjugate is 3rd-order smooth, for OCO with smooth and self-concordant loss functions (such as linear or quadratic losses). We instantiate our algorithm with different regularizers and show that, for example, when the decision set is the $\ell_2$ ball, our algorithm achieves $\tilde{\mathcal{O}}(T^{\frac{2}{5}})$ alternating regret with no dimension dependence (and a better $\tilde{\mathcal{O}}(T^{\frac{1}{3}})$ bound for quadratic losses). We complement our results by showing some algorithm-specific alternating regret lower bounds, including a somewhat surprising $\Omega(\sqrt{T})$ lower bound for a Regret Matching variant that is widely used in alternating learning dynamics.