Trading Regret for Efficiency: Online Convex Optimization with Long Term Constraints

TL;DR

This paper introduces a new framework for online convex optimization that relaxes the need for immediate feasibility, focusing on long-term constraint satisfaction, and offers algorithms with sublinear regret and constraint violation bounds.

Contribution

It proposes an efficient online convex optimization framework that ensures long-term constraint satisfaction with improved regret bounds, using novel algorithms based on convex-concave optimization and Mirror Prox.

Findings

Achieves A(\, ext{T}) ext{ regret} and A(T^{3/4}) ext{ constraint violation} with the first algorithm.

Guarantees long-term constraint satisfaction at the expense of higher regret bounds.

Extends to partial feedback scenarios with a bandit feedback algorithm achieving similar bounds.

Abstract

In this paper we propose a framework for solving constrained online convex optimization problem. Our motivation stems from the observation that most algorithms proposed for online convex optimization require a projection onto the convex set $\mathcal{K}$ from which the decisions are made. While for simple shapes (e.g. Euclidean ball) the projection is straightforward, for arbitrary complex sets this is the main computational challenge and may be inefficient in practice. In this paper, we consider an alternative online convex optimization problem. Instead of requiring decisions belong to $\mathcal{K}$ for all rounds, we only require that the constraints which define the set $\mathcal{K}$ be satisfied in the long run. We show that our framework can be utilized to solve a relaxed version of online learning with side constraints addressed in \cite{DBLP:conf/colt/MannorT06} and \cite{DBLP:conf/aaai/KvetonYTM08}. By turning the problem into an online convex-concave optimization problem, we propose an efficient algorithm which achieves $\tilde{\mathcal{O}}(\sqrt{T})$ regret bound and $\tilde{\mathcal{O}}(T^{3/4})$ bound for the violation of constraints. Then we modify the algorithm in order to guarantee that the constraints are satisfied in the long run. This gain is achieved at the price of getting $\tilde{\mathcal{O}}(T^{3/4})$ regret bound. Our second algorithm is based on the Mirror Prox method \citep{nemirovski-2005-prox} to solve variational inequalities which achieves $\tilde{\mathcal{\mathcal{O}}}(T^{2/3})$ bound for both regret and the violation of constraints when the domain $\K$ can be described by a finite number of linear constraints. Finally, we extend the result to the setting where we only have partial access to the convex set $\mathcal{K}$ and propose a multipoint bandit feedback algorithm with the same bounds in expectation as our first algorithm.