MM Algorithms for Minimizing Nonsmoothly Penalized Objective Functions

TL;DR

This paper introduces a flexible MM algorithm framework for optimizing nonsmoothly penalized functions, emphasizing efficiency, stability, and convergence, with demonstrated effectiveness on simulations and microarray data.

Contribution

It develops a general MM-based optimization framework for nonsmooth penalties, featuring componentwise soft-thresholding and new acceleration techniques, with proven convergence under weaker conditions.

Findings

Algorithms are fast and stable, avoiding high-dimensional matrix inversions.

Acceleration methods significantly improve convergence speed.

Effective on both simulated and real microarray data.

Abstract

In this paper, we propose a general class of algorithms for optimizing an extensive variety of nonsmoothly penalized objective functions that satisfy certain regularity conditions. The proposed framework utilizes the majorization-minimization (MM) algorithm as its core optimization engine. The resulting algorithms rely on iterated soft-thresholding, implemented componentwise, allowing for fast, stable updating that avoids the need for any high-dimensional matrix inversion. We establish a local convergence theory for this class of algorithms under weaker assumptions than previously considered in the statistical literature. We also demonstrate the exceptional effectiveness of new acceleration methods, originally proposed for the EM algorithm, in this class of problems. Simulation results and a microarray data example are provided to demonstrate the algorithm's capabilities and versatility.