Quasi-Newton acceleration of EM and MM algorithms via Broyden$'$s method

TL;DR

This paper introduces a rigorous quasi-Newton acceleration method for MM algorithms that improves convergence speed without requiring gradient information, validated through extensive numerical experiments.

Contribution

It presents a novel, general quasi-Newton approach for accelerating MM algorithms, with convergence guarantees and high-dimensional efficiency.

Findings

Achieves state-of-the-art performance in diverse problems

Converges linearly or super-linearly under certain conditions

Does not require gradient or objective function derivatives

Abstract

The principle of majorization-minimization (MM) provides a general framework for eliciting effective algorithms to solve optimization problems. However, they often suffer from slow convergence, especially in large-scale and high-dimensional data settings. This has drawn attention to acceleration schemes designed exclusively for MM algorithms, but many existing designs are either problem-specific or rely on approximations and heuristics loosely inspired by the optimization literature. We propose a novel, rigorous quasi-Newton method for accelerating any valid MM algorithm, cast as seeking a fixed point of the MM \textit{algorithm map}. The method does not require specific information or computation from the objective function or its gradient and enjoys a limited-memory variant amenable to efficient computation in high-dimensional settings. By connecting our approach to Broyden's classical root-finding methods, we establish convergence guarantees and identify conditions for linear and super-linear convergence. These results are validated numerically and compared to peer methods in a thorough empirical study, showing that it achieves state-of-the-art performance across a diverse range of problems.