Linear Convergence of the Primal-Dual Gradient Method for Convex-Concave Saddle Point Problems without Strong Convexity

TL;DR

This paper proves that the primal-dual gradient method achieves linear convergence for convex-concave saddle point problems with certain conditions, even without strong convexity of the primal function, using a novel analysis technique.

Contribution

It introduces a new analysis method showing linear convergence of the primal-dual gradient method without requiring strong convexity of the primal function.

Findings

Primal-dual gradient method converges linearly under full column rank A.

The analysis applies even when f is not strongly convex.

Extension to stochastic variance reduced gradient (SVRG) method for finite-sum problems.

Abstract

We consider the convex-concave saddle point problem $\min_{x}\max_{y} f(x)+y^\top A x-g(y)$ where $f$ is smooth and convex and $g$ is smooth and strongly convex. We prove that if the coupling matrix $A$ has full column rank, the vanilla primal-dual gradient method can achieve linear convergence even if $f$ is not strongly convex. Our result generalizes previous work which either requires $f$ and $g$ to be quadratic functions or requires proximal mappings for both $f$ and $g$. We adopt a novel analysis technique that in each iteration uses a "ghost" update as a reference, and show that the iterates in the primal-dual gradient method converge to this "ghost" sequence. Using the same technique we further give an analysis for the primal-dual stochastic variance reduced gradient (SVRG) method for convex-concave saddle point problems with a finite-sum structure.