A Primal-Dual Type Algorithm with the $O(1/t)$ Convergence Rate for Large Scale Constrained Convex Programs

TL;DR

This paper introduces a new primal-dual algorithm for large scale constrained convex programs that achieves an improved $O(1/t)$ convergence rate using simple gradient updates, overcoming limitations of previous methods.

Contribution

The paper proposes a novel primal-dual algorithm with $O(1/t)$ convergence rate that is computationally simple and effective for large scale convex programs without separability.

Findings

Achieves $O(1/t)$ convergence rate for large scale problems.

Requires only simple gradient updates per iteration.

Outperforms traditional methods with slower convergence rates.

Abstract

This paper considers large scale constrained convex programs, which are usually not solvable by interior point methods or other Newton-type methods due to the prohibitive computation and storage complexity for Hessians and matrix inversions. Instead, large scale constrained convex programs are often solved by gradient based methods or decomposition based methods. The conventional primal-dual subgradient method, aka, Arrow-Hurwicz-Uzawa subgradient method, is a low complexity algorithm with the $O(1/\sqrt{t})$ convergence rate, where $t$ is the number of iterations. If the objective and constraint functions are separable, the Lagrangian dual type method can decompose a large scale convex program into multiple parallel small scale convex programs. The classical dual gradient algorithm is an example of Lagrangian dual type methods and has convergence rate $O(1/\sqrt{t})$. Recently, a new Lagrangian dual type algorithm with faster $O(1/t)$ convergence is proposed in Yu and Neely (2015). However, if the objective or constraint functions are not separable, each iteration of the Lagrangian dual type method in Yu and Neely (2015) requires to solve a large scale unconstrained convex program, which can have huge complexity. This paper proposes a new primal-dual type algorithm, which only involves simple gradient updates at each iteration and has the $O(1/t)$ convergence rate.