Preconditioned Proximal Gradient Methods with Conjugate Momentum: A Subspace Perspective

TL;DR

This paper introduces a novel subspace proximal Newton method with conjugate momentum for composite optimization, leveraging structure-exploiting preconditioners and low-dimensional subspace curvature information to enhance convergence on high-dimensional, ill-conditioned problems.

Contribution

It develops a subspace proximal Newton framework that combines preconditioning, momentum, and orthogonalization to improve convergence efficiency in composite optimization.

Findings

Global convergence is established.

Achieves Q-linear convergence under strong convexity.

Numerical experiments show superior performance on high-dimensional problems.

Abstract

In this paper, we propose a descent method for composite optimization problems with linear operators. Specifically, we first design a structure-exploiting preconditioner tailored to the linear operator so that the resulting preconditioned proximal subproblem admits a closed-form solution through its dual formulation. However, such a structure-driven preconditioner may be poorly aligned with the local curvature of the smooth component, which can lead to slow practical convergence. To address this issue, we develop a subspace proximal Newton framework that incorporates curvature information within a low-dimensional subspace. At each iteration, the search direction is obtained by minimizing a proximal Newton model restricted to a two-dimensional subspace spanned by the current preconditioned proximal gradient direction and a momentum direction derived from the previous iterate. By orthogonalizing the subspace basis with respect to the local Hessian-induced metric, the resulting two-dimensional nonsmooth subproblem can be efficiently approximated by solving two one-dimensional optimization problems. This orthogonalization plays a crucial role: it allows a single pass of alternating one-dimensional updates to provide a good approximation to the original coupled two-dimensional subproblem while keeping the per-iteration computational cost low. We establish global convergence of the proposed method and prove a $Q$-linear convergence rate under strong convexity. Comparative numerical experiments demonstrate the effectiveness of the proposed algorithm, particularly on high-dimensional and ill-conditioned problems.