Accelerating Multi-Block Constrained Optimization Through Learning to Optimize

TL;DR

This paper introduces a learning-based method to adaptively optimize penalty parameters in multi-block ADMM-type algorithms, significantly improving convergence and performance in constrained optimization problems.

Contribution

It extends Learning to Optimize to multi-block ADMM methods by adaptively tuning penalty parameters via supervised learning, enhancing convergence and efficiency.

Findings

Outperforms existing methods on Lasso and optimal transport problems.

Demonstrates improved convergence and solution quality.

Applicable to a broad class of linearly constrained problems.

Abstract

Learning to Optimize (L2O) approaches, including algorithm unrolling, plug-and-play methods, and hyperparameter learning, have garnered significant attention and have been successfully applied to the Alternating Direction Method of Multipliers (ADMM) and its variants. However, the natural extension of L2O to multi-block ADMM-type methods remains largely unexplored. Such an extension is critical, as multi-block methods leverage the separable structure of optimization problems, offering substantial reductions in per-iteration complexity. Given that classical multi-block ADMM does not guarantee convergence, the Majorized Proximal Augmented Lagrangian Method (MPALM), which shares a similar form with multi-block ADMM and ensures convergence, is more suitable in this setting. Despite its theoretical advantages, MPALM's performance is highly sensitive to the choice of penalty parameters. To address this limitation, we propose a novel L2O approach that adaptively selects this hyperparameter using supervised learning. We demonstrate the versatility and effectiveness of our method by applying it to the Lasso problem and the optimal transport problem. Our numerical results show that the proposed framework outperforms popular alternatives. Given its applicability to generic linearly constrained composite optimization problems, this work opens the door to a wide range of potential real-world applications.