Can a Learner Regret Using a No-Regret Algorithm? A Control-Theoretic Study of Performance Dominance

TL;DR

This paper investigates whether some no-regret learning algorithms can outperform others across all environments, revealing that anticipatory replicator dynamics can globally dominate standard no-regret algorithms, thus challenging the notion of a 'free-lunch' in regret minimization.

Contribution

The paper introduces a control-theoretic framework to compare no-regret algorithms, demonstrating that anticipatory replicator dynamics can globally outperform standard no-regret methods.

Findings

Anticipatory RD globally dominates standard RD in all payoff environments.

A passivity-based approach is used for performance comparison.

Optimal control formulation shows zero minimal reward gap.

Abstract

No-regret learning dynamics ensure that a learner asymptotically achieves an average reward no worse than that of any fixed strategy. This no-regret guarantee does not determine the value of the asymptotic average reward. Indeed, it is possible for different no-regret learning dynamics to exhibit different asymptotic average rewards when facing the same environment while both assure the no-regret guarantee. This paper asks whether a "free-lunch" phenomenon can arise among no-regret algorithms. Namely, is it possible for one no-regret learning rule to uniformly outperform another no-regret learning rule across all payoff environments. Stated differently, can a learner regret not using a particular no-regret algorithm? We consider generalized replicator dynamics (RD) as a cascade interconnection between a linear time-invariant (LTI) system and the softmax nonlinearity. Varying this LTI system leads to different realizations of replicator dynamics, including so-called anticipatory RD, exponential RD, and other forms of higher-order RD. Setting the LTI system to be an integrator realizes standard RD, which is known to satisfy the no-regret property. Within this framework, we analyze and compare various realizations of these generalized realizations RD by varying the LTI system. We first formulate performance comparison as a passivity property of an associated comparison system and establish "local" dominance results, i.e., comparing the asymptotic performance near an equilibrium payoff vector. We then cast performance comparison between a form of anticipatory RD and standard RD as an optimal-control problem. We show that the minimal achievable cumulative reward gap is zero, thereby establishing global dominance of anticipatory RD across all payoff environments and establishing a "free lunch" among no-regret learning dynamics.