Linear Convergence in Games with Delayed Feedback via Extra Prediction

TL;DR

This paper analyzes the impact of delayed feedback on convergence in bilinear games and demonstrates that extra optimism in prediction significantly accelerates convergence rates, offering a promising approach to mitigate delay effects.

Contribution

It introduces a novel analysis of WOGDA with extra optimism, showing improved convergence rates under feedback delays in bilinear games.

Findings

Extra optimism accelerates convergence rate to $ ext{exp}(- heta(t/(m^{2} ext{log} m)))$

Standard optimism achieves linear convergence at rate $ ext{exp}(- heta(t/m^{5}))$

Experiments confirm the theoretical acceleration due to extra optimism.

Abstract

Feedback delays are inevitable in real-world multi-agent learning. They are known to severely degrade performance, and the convergence rate under delayed feedback is still unclear, even for bilinear games. This paper derives the rate of linear convergence of Weighted Optimistic Gradient Descent-Ascent (WOGDA), which predicts future rewards with extra optimism, in unconstrained bilinear games. To analyze the algorithm, we interpret it as an approximation of the Extra Proximal Point (EPP), which is updated based on farther future rewards than the classical Proximal Point (PP). Our theorems show that standard optimism (predicting the next-step reward) achieves linear convergence to the equilibrium at a rate $\exp(-\Theta(t/m^{5}))$ after $t$ iterations for delay $m$. Moreover, employing extra optimism (predicting farther future reward) tolerates a larger step size and significantly accelerates the rate to $\exp(-\Theta(t/(m^{2}\log m)))$. Our experiments also show accelerated convergence driven by the extra optimism and are qualitatively consistent with our theorems. In summary, this paper validates that extra optimism is a promising countermeasure against performance degradation caused by feedback delays.