A new insight on augmented Lagrangian method with applications in machine learning

TL;DR

This paper introduces a novel relaxed augmented Lagrangian method with double-penalty terms for convex optimization, demonstrating superior performance in machine learning tasks like SVM and PCA.

Contribution

It develops a new augmented Lagrangian approach with double penalties, extending to multi-block problems and providing convergence analysis and practical algorithms.

Findings

Algorithms outperform state-of-the-art methods in SVM and PCA tasks.

Convergence rates are established for the proposed methods.

Extends to multi-block separable convex optimization problems.

Abstract

By exploiting double-penalty terms for the primal subproblem, we develop a novel relaxed augmented Lagrangian method for solving a family of convex optimization problems subject to equality or inequality constraints. The method is then extended to solve a general multi-block separable convex optimization problem, and two related primal-dual hybrid gradient algorithms are also discussed. Convergence results about the sublinear and linear convergence rates are established by variational characterizations for both the saddle-point of the problem and the first-order optimality conditions of involved subproblems. A large number of experiments on testing the linear support vector machine problem and the robust principal component analysis problem arising from machine learning indicate that our proposed algorithms perform much better than several state-of-the-art algorithms.