An Accelerated Block Proximal Framework with Adaptive Momentum for Nonconvex and Nonsmooth Optimization

TL;DR

This paper introduces an accelerated block proximal linear framework with adaptive momentum (ABPL$^+$) for nonconvex, nonsmooth optimization, featuring adaptive extrapolation, random block updates, and proven convergence properties, validated through extensive experiments.

Contribution

It presents a novel ABPL$^+$ algorithm with adaptive momentum, random block updating, and convergence guarantees for nonconvex nonsmooth problems, extending existing methods with improved flexibility and performance.

Findings

The algorithm monotonically decreases the objective function value.

ABPL$^+$ achieves linear and sublinear convergence rates under KL conditions.

Numerical experiments confirm the effectiveness in matrix and tensor decomposition tasks.

Abstract

We propose an accelerated block proximal linear framework with adaptive momentum (ABPL$^+$) for nonconvex and nonsmooth optimization. We analyze the potential causes of the extrapolation step failing in some algorithms, and resolve this issue by enhancing the comparison process that evaluates the trade-off between the proximal gradient step and the linear extrapolation step in our algorithm. Furthermore, we extends our algorithm to any scenario involving updating block variables with positive integers, allowing each cycle to randomly shuffle the update order of the variable blocks. Additionally, under mild assumptions, we prove that ABPL$^+$ can monotonically decrease the function value without strictly restricting the extrapolation parameters and step size, demonstrates the viability and effectiveness of updating these blocks in a random order, and we also more obviously and intuitively demonstrate that the derivative set of the sequence generated by our algorithm is a critical point set. Moreover, we demonstrate the global convergence as well as the linear and sublinear convergence rates of our algorithm by utilizing the Kurdyka-Lojasiewicz (K{\L}) condition. To enhance the effectiveness and flexibility of our algorithm, we also expand the study to the imprecise version of our algorithm and construct an adaptive extrapolation parameter strategy, which improving its overall performance. We apply our algorithm to multiple non-negative matrix factorization with the $\ell_0$ norm, nonnegative tensor decomposition with the $\ell_0$ norm, and perform extensive numerical experiments to validate its effectiveness and efficiency.