MM for Penalized Estimation

TL;DR

This paper explores the use of the majorization-minimization (MM) algorithm for penalized estimation with both convex and nonconvex loss functions, emphasizing robustness to outliers and providing convergence theory.

Contribution

It introduces novel optimality conditions and convergence results for MM algorithms applied to penalized estimation with nonconvex loss functions, expanding their theoretical foundation.

Findings

Algorithms perform well on simulated data

Effective on real healthcare and cancer data

Available in R package mpath

Abstract

Penalized estimation can conduct variable selection and parameter estimation simultaneously. The general framework is to minimize a loss function subject to a penalty designed to generate sparse variable selection. The majorization-minimization (MM) algorithm is a computational scheme for stability and simplicity, and the MM algorithm has been widely applied in penalized estimation. Much of the previous work have focused on convex loss functions such as generalized linear models. When data are contaminated with outliers, robust loss functions can generate more reliable estimates. Recent literature has witnessed a growing impact of nonconvex loss-based methods, which can generate robust estimation for data contaminated with outliers. This article investigates MM algorithm for penalized estimation, provide innovative optimality conditions and establish convergence theory with both convex and nonconvex loss functions. With respect to applications, we focus on several nonconvex loss functions, which were formerly studied in machine learning for regression and classification problems. Performance of the proposed algorithms are evaluated on simulated and real data including healthcare costs and cancer clinical status. Efficient implementations of the algorithms are available in the R package mpath in CRAN.