An Introduction to MM Algorithms for Machine Learning and Statistical

TL;DR

This paper introduces the MM algorithm framework, illustrating its application in machine learning and statistical estimation through examples like Gaussian mixture regressions, multinomial logistic regressions, and support vector machines, including derivations and demonstrations.

Contribution

It provides a comprehensive overview of MM algorithms, detailing their design, derivation for specific models, and practical implementation in machine learning tasks.

Findings

Effective algorithms for Gaussian mixture regressions

Derived MM algorithms for multinomial logistic regressions

Numerical demonstrations validate the approaches

Abstract

MM (majorization--minimization) algorithms are an increasingly popular tool for solving optimization problems in machine learning and statistical estimation. This article introduces the MM algorithm framework in general and via three popular example applications: Gaussian mixture regressions, multinomial logistic regressions, and support vector machines. Specific algorithms for the three examples are derived and numerical demonstrations are presented. Theoretical and practical aspects of MM algorithm design are discussed.