Lagrangian-based methods in convex optimization: prediction-correction frameworks with non-ergodic convergence rates

TL;DR

This paper introduces new prediction-correction frameworks for Lagrangian-based convex optimization methods, achieving improved non-ergodic convergence rates, including $O(1/k)$ and $O(1/k^2)$, with applications to ADMM variants.

Contribution

It proposes novel prediction-correction frameworks that enhance convergence rates of Lagrangian-based methods, including ADMM, under general convexity and strong convexity assumptions.

Findings

Achieves $O(1/k)$ non-ergodic convergence rate for general convex problems.

Achieves $O(1/k^2)$ non-ergodic convergence rate under strong convexity or Lipschitz conditions.

Establishes convergence rates for well-known Lagrangian-based methods like ADMM.

Abstract

Lagrangian-based methods are classical methods for solving convex optimization problems with equality constraints. We present novel prediction-correction frameworks for such methods and their variants, which can achieve $O(1/k)$ non-ergodic convergence rates for general convex optimization and $O(1/k^2)$ non-ergodic convergence rates under the assumption that the objective function is strongly convex or gradient Lipschitz continuous. We give two approaches ($updating~multiplier~once$ $or~twice$) to design algorithms satisfying the presented prediction-correction frameworks. As applications, we establish non-ergodic convergence rates for some well-known Lagrangian-based methods (esp., the ADMM type methods and the multi-block ADMM type methods).