Negative Momentum for Convex-Concave Optimization

TL;DR

This paper demonstrates that negative momentum can achieve global convergence and accelerated rates in convex-concave min-max optimization, challenging previous beliefs about its limitations.

Contribution

It proves that negative momentum enables both global convergence and faster rates in convex-concave settings, broadening its applicability.

Findings

Negative momentum achieves global convergence in convex-concave optimization.

Negative momentum enables accelerated convergence in strongly-convex-strong-concave problems.

Results show negative momentum is more effective and general than previously believed.

Abstract

This paper revisits momentum in the context of min-max optimization. Momentum is a celebrated mechanism for accelerating gradient dynamics in settings like convex minimization, but its direct use in min-max optimization makes gradient dynamics diverge. Surprisingly, Gidel et al. 2019 showed that negative momentum can help fix convergence. However, despite these promising initial results and progress since, the power of momentum remains unclear for min-max optimization in two key ways. (1) Generality: is global convergence possible for the foundational setting of convex-concave optimization? This is the direct analog of convex minimization and is a standard testing ground for min-max algorithms. (2) Fast convergence: is accelerated convergence possible for strongly-convex-strong-concave optimization (the only non-linear setting where global convergence is known)? Recent work has even argued that this is impossible. We answer both these questions in the affirmative. Together, these results put negative momentum on more equal footing with competitor algorithms, and show that negative momentum enables convergence significantly faster and more generally than was known possible.