Matrix-Free Solvers for Exact Penalty Subproblems

TL;DR

This paper introduces two matrix-free algorithms, IRWA and ADAL, for efficiently solving large-scale exact penalty subproblems with proven convergence and practical effectiveness demonstrated through numerical experiments.

Contribution

The paper presents novel matrix-free iterative re-weighting and ADAL-based methods for large-scale optimization, with convergence guarantees and improved reliability of IRWA in certain cases.

Findings

Both algorithms are globally convergent.

Each algorithm requires at most O(1/ε^2) iterations.

Numerical experiments show efficient inexact solutions, with IRWA sometimes more reliable than ADAL.

Abstract

We present two matrix-free methods for approximately solving exact penalty subproblems that arise when solving large-scale optimization problems. The first approach is a novel iterative re-weighting algorithm (IRWA), which iteratively minimizes quadratic models of relaxed subproblems while automatically updating a relaxation vector. The second approach is based on alternating direction augmented Lagrangian (ADAL) technology applied to our setting. The main computational costs of each algorithm are the repeated minimizations of convex quadratic functions which can be performed matrix-free. We prove that both algorithms are globally convergent under loose assumptions, and that each requires at most $O(1/\varepsilon^2)$ iterations to reach $\varepsilon$-optimality of the objective function. Numerical experiments exhibit the ability of both algorithms to efficiently find inexact solutions. Moreover, in certain cases, IRWA is shown to be more reliable than ADAL.