Multi-Model Subset Selection

TL;DR

This paper introduces a novel multi-model ensemble regression method that combines the interpretability of sparse models with the high accuracy of ensemble techniques, supported by an efficient optimization algorithm and empirical validation.

Contribution

It proposes a joint optimization algorithm for multi-model L0-penalized regression ensembles, extending sparse methods to improve prediction accuracy while maintaining interpretability.

Findings

Ensembles outperform state-of-the-art methods on simulated data.

The method provides interpretable models with high prediction accuracy.

The approach is supported by theoretical insights into bias, variance, and variable selection.

Abstract

The two primary approaches for high-dimensional regression problems are sparse methods (e.g., best subset selection, which uses the L0-norm in the penalty) and ensemble methods (e.g., random forests). Although sparse methods typically yield interpretable models, in terms of prediction accuracy they are often outperformed by "blackbox" multi-model ensemble methods. A regression ensemble is introduced which combines the interpretability of sparse methods with the high prediction accuracy of ensemble methods. An algorithm is proposed to solve the joint optimization of the corresponding L0-penalized regression models by extending recent developments in L0-optimization for sparse methods to multi-model regression ensembles. The sparse and diverse models in the ensemble are learned simultaneously from the data. Each of these models provides an explanation for the relationship between a subset of predictors and the response variable. Empirical studies and theoretical knowledge about ensembles are used to gain insight into the ensemble method's performance, focusing on the interplay between bias, variance, covariance, and variable selection. In prediction tasks, the ensembles can outperform state-of-the-art competitors on both simulated and real data. Forward stepwise regression is also generalized to multi-model regression ensembles and used to obtain an initial solution for the algorithm. The optimization algorithms are implemented in publicly available software packages.