A Controllability Perspective on Steering Follow-the-Regularized-Leader Learners in Games

TL;DR

This paper studies how a controller can steer learners in game dynamics modeled as nonlinear control systems, providing conditions for controllability in multi-agent FTRL settings with applications to classic examples.

Contribution

It introduces a controllability framework for FTRL learners in games, offering necessary and sufficient conditions for steering strategies in two-player and multi-learner scenarios.

Findings

Necessary and sufficient controllability criterion for two-player case.

Sufficient controllability conditions for multi-learner interactions.

Application of results to Rock-Paper-Scissors and Brockett's integrator.

Abstract

Follow-the-regularized-leader (FTRL) algorithms have become popular in the context of games, providing easy-to-implement methods for each agent, as well as theoretical guarantees that the strategies of all agents will converge to some equilibrium concept (provided that all agents follow the appropriate dynamics). However, with these methods, each agent ignores the coupling in the game, and treats their payoff vectors as exogenously given. In this paper, we take the perspective of one agent (the controller) deciding their mixed strategies in a finite game, while one or more other agents update their mixed strategies according to continuous-time FTRL. Viewing the learners' dynamics as a nonlinear control system evolving on the relative interior of a simplex or product of simplices, we ask when the controller can steer the learners to a target state, using only its own mixed strategy and without modifying the game's payoff structure. For the two-player case we provide a necessary and sufficient criterion for controllability based on the existence of a fully mixed neutralizing controller strategy and a rank condition on the projected payoff map. For multi-learner interactions we give two sufficient controllability conditions, one based on uniform neutralization and one based on a periodic-drift hypothesis together with a Lie-algebra rank condition. We illustrate these results on canonical examples such as Rock-Paper-Scissors and a construction related to Brockett's integrator.