An inexact proximal augmented Lagrangian framework with arbitrary linearly convergent inner solver for composite convex optimization

TL;DR

This paper introduces an inexact proximal augmented Lagrangian method with adaptive stopping criteria for composite convex optimization, accommodating arbitrary linearly convergent inner solvers, including stochastic algorithms, to improve scalability and efficiency.

Contribution

It develops a flexible framework with explicit inexact inner problem solutions and provides complexity bounds under various assumptions, enhancing scalability for large-scale convex optimization.

Findings

Achieves $O(1/ oot ext{epsilon})$ complexity bounds for bounded domains.

Extends bounds to unbounded domains with logarithmic factors.

Demonstrates potential speedup with randomized inner solvers in experiments.

Abstract

We propose an inexact proximal augmented Lagrangian framework with explicit inner problem termination rule for composite convex optimization problems. We consider arbitrary linearly convergent inner solver including in particular stochastic algorithms, making the resulting framework more scalable facing the ever-increasing problem dimension. Each subproblem is solved inexactly with an explicit and self-adaptive stopping criterion, without requiring to set an a priori target accuracy. When the primal and dual domain are bounded, our method achieves $O(1/\sqrt{\epsilon})$ and $O(1/{\epsilon})$ complexity bound in terms of number of inner solver iterations, respectively for the strongly convex and non-strongly convex case. Without the boundedness assumption, only logarithm terms need to be added and the above two complexity bounds increase respectively to $\tilde O(1/\sqrt{\epsilon})$ and $\tilde O(1/{\epsilon})$, which hold both for obtaining $\epsilon$-optimal and $\epsilon$-KKT solution. Within the general framework that we propose, we also obtain $\tilde O(1/{\epsilon})$ and $\tilde O(1/{\epsilon^2})$ complexity bounds under relative smoothness assumption on the differentiable component of the objective function. We show through theoretical analysis as well as numerical experiments the computational speedup possibly achieved by the use of randomized inner solvers for large-scale problems.