L1-Penalization for Mixture Regression Models

TL;DR

This paper introduces an l1-penalized maximum likelihood estimator for high-dimensional finite mixture of regressions models, addressing non-convex optimization challenges and enabling variable selection with proven convergence and theoretical guarantees.

Contribution

It develops an efficient EM algorithm for penalized mixture regression, providing a novel approach to regularization and variable selection in non-convex high-dimensional models.

Findings

Numerical stability improved with penalization

Effective variable selection demonstrated

Convergence properties established

Abstract

We consider a finite mixture of regressions (FMR) model for high-dimensional inhomogeneous data where the number of covariates may be much larger than sample size. We propose an l1-penalized maximum likelihood estimator in an appropriate parameterization. This kind of estimation belongs to a class of problems where optimization and theory for non-convex functions is needed. This distinguishes itself very clearly from high-dimensional estimation with convex loss- or objective functions, as for example with the Lasso in linear or generalized linear models. Mixture models represent a prime and important example where non-convexity arises. For FMR models, we develop an efficient EM algorithm for numerical optimization with provable convergence properties. Our penalized estimator is numerically better posed (e.g., boundedness of the criterion function) than unpenalized maximum likelihood estimation, and it allows for effective statistical regularization including variable selection. We also present some asymptotic theory and oracle inequalities: due to non-convexity of the negative log-likelihood function, different mathematical arguments are needed than for problems with convex losses. Finally, we apply the new method to both simulated and real data.