Finite-Time Last-Iterate Convergence for Multi-Agent Learning in Games

TL;DR

This paper establishes finite-time last-iterate convergence rates for multi-agent online gradient descent in $\lambda$-cocoercive games, introduces a fully adaptive algorithm, and extends results to noisy feedback scenarios, filling key gaps in the literature.

Contribution

It provides the first finite-time convergence analysis for adaptive multi-agent learning algorithms in $\lambda$-cocoercive games, including noisy feedback cases.

Findings

Finite-time last-iterate convergence rates established.

Adaptive OGD algorithm matches non-adaptive convergence.

Results extend to noisy gradient feedback with non-decreasing step-sizes.

Abstract

In this paper, we consider multi-agent learning via online gradient descent in a class of games called $\lambda$-cocoercive games, a fairly broad class of games that admits many Nash equilibria and that properly includes unconstrained strongly monotone games. We characterize the finite-time last-iterate convergence rate for joint OGD learning on $\lambda$-cocoercive games; further, building on this result, we develop a fully adaptive OGD learning algorithm that does not require any knowledge of problem parameter (e.g. cocoercive constant $\lambda$) and show, via a novel double-stopping time technique, that this adaptive algorithm achieves same finite-time last-iterate convergence rate as non-adaptive counterpart. Subsequently, we extend OGD learning to the noisy gradient feedback case and establish last-iterate convergence results -- first qualitative almost sure convergence, then quantitative finite-time convergence rates -- all under non-decreasing step-sizes. To our knowledge, we provide the first set of results that fill in several gaps of the existing multi-agent online learning literature, where three aspects -- finite-time convergence rates, non-decreasing step-sizes, and fully adaptive algorithms have been unexplored before.