Hierarchies of No-regret Algorithms

TL;DR

This paper investigates the hierarchy of no-regret algorithms in two-player games, revealing counterintuitive utility outcomes and the impact of learning rates on algorithm performance.

Contribution

It uncovers the surprising result that no-swap-regret algorithms can be worse for players and introduces methods to balance learning rates for fairer comparisons.

Findings

No-swap-regret algorithms can yield higher opponent utility in many games.

Slower learning rates of no-swap-regret algorithms explain their poorer performance.

In certain random games, no-swap-regret algorithms outperform no-regret algorithms.

Abstract

Our paper studies the setting of players using no-regret algorithms in various two-player games. We address whether having stronger regret guarantees or playing against an opponent with weaker regret guarantees yields higher utilities for the player in question. We consider a hierarchy of algorithms from weakest to strongest: uniform random play, no-regret, and no-swap-regret. We find, counterintuitively, that in many games, no-swap-regret is a worse choice for players (and gives better utility for their opponents). We find the root cause of this phenomenon to be a difference in effective learning rate between the two algorithms, where the no-swap-regret algorithms learn $N$ times slower than no-regret algorithms. To address this, we attempt to equalize learning rates, leading to closer utility between no-regret and no-swap-regret players. Finally, we show that for certain random games with $7$ actions per player, no-swap-regret algorithms can perform noticeably better than no-regret algorithms in a manner that cannot be explained away by unfairly adjusted learning rates.