A Unified Primal-Dual Algorithm Framework for Inequality Constrained Problems

TL;DR

This paper introduces a versatile primal-dual algorithm framework for inequality constrained convex problems, unifying existing methods and proposing a new efficient algorithm with improved convergence and practical performance.

Contribution

It unifies various primal-dual algorithms under a single framework, introduces Simi-OGDA, and analyzes the impact of penalty terms on convergence.

Findings

The framework achieves $ ext{O}(1/N)$ ergodic convergence rate.

Proper penalty selection enhances numerical performance and convergence.

Numerical experiments demonstrate the efficiency of the proposed methods.

Abstract

In this paper, we propose a unified primal-dual algorithm framework based on the augmented Lagrangian function for composite convex problems with conic inequality constraints. The new framework is highly versatile. First, it not only covers many existing algorithms such as PDHG, Chambolle-Pock (CP), GDA, OGDA and linearized ALM, but also guides us to design a new efficient algorithm called Simi-OGDA (SOGDA). Second, it enables us to study the role of the augmented penalty term in the convergence analysis. Interestingly, a properly selected penalty not only improves the numerical performance of the above methods, but also theoretically enables the convergence of algorithms like PDHG and SOGDA. Under properly designed step sizes and penalty term, our unified framework preserves the $\mathcal{O}(1/N)$ ergodic convergence while not requiring any prior knowledge about the magnitude of the optimal Lagrangian multiplier. Linear convergence rate for affine equality constrained problem is also obtained given appropriate conditions. Finally, numerical experiments on linear programming, $\ell_1$ minimization problem, and multi-block basis pursuit problem demonstrate the efficiency of our methods.