Connections between convex optimization algorithms and subspace correction methods

TL;DR

This paper establishes a unified framework linking convex optimization algorithms with subspace correction methods through duality, enabling the systematic development of new algorithms and enhancing understanding of existing ones.

Contribution

It introduces the concept of dualization to connect various convex optimization algorithms with subspace correction methods, leading to new algorithmic variants and convergence guarantees.

Findings

Classical algorithms are shown as dualizations of subspace correction methods.

Parallel variants of algorithms are derived for large-scale problems.

New ADMM-type algorithms ensure convergence in multi-block settings.

Abstract

We show that a broad range of convex optimization algorithms, including alternating projection, operator splitting, and multiplier methods, can be systematically derived from the framework of subspace correction methods via convex duality. To formalize this connection, we introduce the notion of dualization, a process that transforms an iterative method for the dual problem into an equivalent method for the primal problem. This concept establishes new connections across these algorithmic classes, encompassing both well-known and new methods. In particular, we show that classical algorithms such as the von Neumann, Dykstra, Peaceman--Rachford, and Douglas--Rachford methods can be interpreted as dualizations of subspace correction methods applied to appropriate dual formulations. Beyond unifying existing methods, our framework enables the systematic development of new algorithms for convex optimization. For instance, we derive parallel variants of alternating projection and operator splitting methods, as dualizations of parallel subspace correction methods, that are well-suited for large-scale problems on modern computing architectures and offer straightforward convergence guarantees. We also propose new alternating direction method of multipliers-type algorithms, derived as dualizations of certain operator splitting methods. These algorithms naturally ensure convergence even in the multi-block setting, where the conventional method does not guarantee convergence when applied to more than two blocks. This unified perspective not only facilitates algorithm design and the transfer of theoretical results but also opens new avenues for research and innovation in convex optimization.