Universal Majorization-Minimization Algorithms

TL;DR

This paper introduces universal majorization-minimization algorithms that automatically generate majorizers using advanced automatic differentiation, enabling broad applicability and convergence without hyperparameter tuning.

Contribution

It presents a novel approach to automatically derive majorizers for MM algorithms, expanding their applicability to arbitrary problems.

Findings

Applicable to any optimization problem

Converge from any starting point

Require no hyperparameter tuning

Abstract

Majorization-minimization (MM) is a family of optimization methods that iteratively reduce a loss by minimizing a locally-tight upper bound, called a majorizer. Traditionally, majorizers were derived by hand, and MM was only applicable to a small number of well-studied problems. We present optimizers that instead derive majorizers automatically, using a recent generalization of Taylor mode automatic differentiation. These universal MM optimizers can be applied to arbitrary problems and converge from any starting point, with no hyperparameter tuning.