A symmetric primal-dual algorithmic framework for saddle point problems

TL;DR

This paper introduces a symmetric primal-dual algorithmic framework (SPIDA) for convex-concave saddle point problems, unifying and extending existing methods with proven convergence and practical efficiency in image processing and machine learning tasks.

Contribution

The paper presents a novel symmetric primal-dual framework with Bregman regularization, unifying and generalizing several existing algorithms for saddle point problems.

Findings

SPIDA converges globally under mild conditions.

SPIDA achieves linear convergence rate under error bound conditions.

Numerical experiments show SPIDA's effectiveness on real-world datasets.

Abstract

In this paper, we propose a new primal-dual algorithmic framework for a class of convex-concave saddle point problems frequently arising from image processing and machine learning. Our algorithmic framework updates the primal variable between the twice calculations of the dual variable, thereby appearing a symmetric iterative scheme, which is accordingly called the symmetric primal-dual algorithm (SPIDA). It is noteworthy that the subproblems of our SPIDA are equipped with Bregman proximal regularization terms, which make SPIDA versatile in the sense that it enjoys an algorithmic framework to understand the iterative schemes of some existing algorithms, such as the classical augmented Lagrangian method (ALM), linearized ALM, and Jacobian splitting algorithms for linearly constrained optimization problems. Besides, our algorithmic framework allows us to derive some customized versions so that SPIDA works as efficiently as possible for structured optimization problems. Theoretically, under some mild conditions, we prove the global convergence of SPIDA and estimate the linear convergence rate under a generalized error bound condition defined by Bregman distance. Finally, a series of numerical experiments on the basis pursuit, robust principal component analysis, and image restoration demonstrate that our SPIDA works well on synthetic and real-world datasets.