Scalable Subset Selection in Linear Mixed Models

TL;DR

This paper introduces a scalable $ ext{L}_0$ regularized method for subset selection in linear mixed models, capable of handling thousands of predictors efficiently, with proven convergence and strong empirical results.

Contribution

It develops a novel $ ext{L}_0$ regularized approach with algorithms that scale to large datasets, filling a gap in sparse learning for LMMs.

Findings

Method runs in seconds to minutes on datasets with thousands of predictors.

Provides finite-sample bounds on divergence, ensuring statistical reliability.

Demonstrates excellent performance on synthetic and real data.

Abstract

Linear mixed models (LMMs), which incorporate fixed and random effects, are key tools for analyzing heterogeneous data, such as in personalized medicine. Nowadays, this type of data is increasingly wide, sometimes containing thousands of candidate predictors, necessitating sparsity for prediction and interpretation. However, existing sparse learning methods for LMMs do not scale well beyond tens or hundreds of predictors, leaving a large gap compared with sparse methods for linear models, which ignore random effects. This paper closes the gap with a new $\ell_0$ regularized method for LMM subset selection that can run on datasets containing thousands of predictors in seconds to minutes. On the computational front, we develop a coordinate descent algorithm as our main workhorse and provide a guarantee of its convergence. We also develop a local search algorithm to help traverse the nonconvex optimization surface. Both algorithms readily extend to subset selection in generalized LMMs via a penalized quasi-likelihood approximation. On the statistical front, we provide a finite-sample bound on the Kullback-Leibler divergence of the new method. We then demonstrate its excellent performance in experiments involving synthetic and real datasets.