First-order majorization-minimization meets high-order majorant: Boosted inexact high-order forward-backward method

TL;DR

This paper develops a novel high-order majorization-minimization framework using a non-quadratic majorant, leading to an inexact forward-backward algorithm with proven convergence and promising experimental results on inverse problems.

Contribution

It introduces a new high-order majorant framework valid for p-paraconcave functions and develops a boosted inexact high-order forward-backward method with convergence guarantees.

Findings

The proposed HiFBA algorithm converges under inexactness conditions.

Boosted HiFBA accelerates convergence with line-search.

Preliminary experiments show efficiency on inverse problems.

Abstract

This paper introduces a first-order majorization-minimization framework based on a high-order majorant for continuous functions, incorporating a non-quadratic regularization term of degree $p>1$. Notably, it is shown to be valid if and only if the function is $p$-paraconcave, thus extending beyond Lipschitz and H\"{o}lder gradient continuity for $p \in (1,2]$, and implying concavity for $p>2$. In the smooth setting, this majorant recovers a variant of the classical descent lemma with quadratic regularization. Building on this foundation, we develop a high-order inexact forward-backward algorithm (HiFBA) and its line-search-accelerated variant, named Boosted HiFBA. For convergence analysis, we introduce a high-order forward-backward envelope (HiFBE), which serves as a Lyapunov function. We establish subsequential convergence under suitable inexactness conditions, and we prove global convergence with linear rates for functions satisfying the Kurdyka-\L{}ojasiewicz inequality. Our preliminary experiments on linear inverse problems and regularized nonnegative matrix factorization highlight the efficiency of HiFBA and its boosted variant, demonstrating their potential for solving challenging nonconvex optimization problems.