Online Stackelberg Optimization via Nonlinear Control

TL;DR

This paper introduces a unified online control framework for Stackelberg optimization in adaptive agent interactions, achieving low regret and broad applicability to problems like pricing, recommendations, and game theory.

Contribution

It develops a tractable algorithmic framework for online nonlinear control with local controllability, extending to adversarial disturbances and bandit feedback scenarios.

Findings

Achieves $O( ootT)$ regret with known dynamics

Provides tight bounds under adversarial disturbances

Demonstrates applications in pricing, recommendations, and game theory

Abstract

In repeated interaction problems with adaptive agents, our objective often requires anticipating and optimizing over the space of possible agent responses. We show that many problems of this form can be cast as instances of online (nonlinear) control which satisfy \textit{local controllability}, with convex losses over a bounded state space which encodes agent behavior, and we introduce a unified algorithmic framework for tractable regret minimization in such cases. When the instance dynamics are known but otherwise arbitrary, we obtain oracle-efficient $O(\sqrt{T})$ regret by reduction to online convex optimization, which can be made computationally efficient if dynamics are locally \textit{action-linear}. In the presence of adversarial disturbances to the state, we give tight bounds in terms of either the cumulative or per-round disturbance magnitude (for \textit{strongly} or \textit{weakly} locally controllable dynamics, respectively). Additionally, we give sublinear regret results for the cases of unknown locally action-linear dynamics as well as for the bandit feedback setting. Finally, we demonstrate applications of our framework to well-studied problems including performative prediction, recommendations for adaptive agents, adaptive pricing of real-valued goods, and repeated gameplay against no-regret learners, directly yielding extensions beyond prior results in each case.