Fast Alternating Linearization Methods for Minimizing the Sum of Two Convex Functions

TL;DR

This paper introduces fast first-order alternating linearization algorithms for efficiently minimizing the sum of two convex functions, with theoretical iteration bounds and practical numerical validation.

Contribution

It develops accelerated alternating linearization methods with improved iteration complexity for convex optimization problems.

Findings

Accelerated methods require at most O(1/√ε) iterations.

Algorithms handle both smooth and one smooth convex function.

Numerical results confirm theoretical efficiency and practical potential.

Abstract

We present in this paper first-order alternating linearization algorithms based on an alternating direction augmented Lagrangian approach for minimizing the sum of two convex functions. Our basic methods require at most $O(1/\epsilon)$ iterations to obtain an $\epsilon$-optimal solution, while our accelerated (i.e., fast) versions of them require at most $O(1/\sqrt{\epsilon})$ iterations, with little change in the computational effort required at each iteration. For both types of methods, we present one algorithm that requires both functions to be smooth with Lipschitz continuous gradients and one algorithm that needs only one of the functions to be so. Algorithms in this paper are Gauss-Seidel type methods, in contrast to the ones proposed by Goldfarb and Ma in [21] where the algorithms are Jacobi type methods. Numerical results are reported to support our theoretical conclusions and demonstrate the practical potential of our algorithms.