On the Convergence of Primal-Dual Proximal Incremental Aggregated Gradient Algorithms

TL;DR

This paper introduces primal-dual proximal incremental aggregated gradient methods for saddle point problems, proving sublinear and linear convergence under various conditions, and extends these methods with and without extrapolation.

Contribution

It adapts proximal incremental aggregated gradient methods to saddle point problems and establishes convergence properties, including linear convergence under strong convexity.

Findings

Primal-dual gap converges sublinearly to 0.

Iteration sequence converges to a saddle point.

Linear convergence under strong convexity conditions.

Abstract

In this paper, we adapt proximal incremental aggregated gradient methods to saddle point problems, which is motivated by decoupling linear transformations in regularized empirical risk minimization models. First, the Primal-Dual Proximal Incremental Aggregated (PD-PIAG) methods with extrapolations were proposed. We proved that the primal-dual gap of the averaged iteration sequence sublinearly converges to 0, and the iteration sequence converges to some saddle point. Under the strong convexity of $f$ and $h^\ast$, we proved that the iteration sequence linearly converges to the saddle point. Then, we propose a PD-PIAG method without extrapolations. The primal-dual gap of the iteration sequence is proved to be sublinearly convergent under strong convexity of $f$.