On the Use of Information Criteria for Subset Selection in Least Squares Regression

TL;DR

This paper introduces BOSS, a scalable orthogonalized subset selection method for linear regression, which uses an information criterion with a heuristic degrees of freedom estimate to efficiently select predictors.

Contribution

The paper proposes BOSS, a novel LS-based subset selection method that scales to large datasets and incorporates a heuristic degrees of freedom for information criteria.

Findings

BOSS outperforms existing LS-based methods in simulations and real data.

BOSS is computationally efficient, requiring only a single LS fit.

The proposed AICc-hdf criterion improves subset selection accuracy.

Abstract

Least squares (LS)-based subset selection methods are popular in linear regression modeling. Best subset selection (BS) is known to be NP-hard and has a computational cost that grows exponentially with the number of predictors. Recently, Bertsimas (2016) formulated BS as a mixed integer optimization (MIO) problem and largely reduced the computation overhead by using a well-developed optimization solver, but the current methodology is not scalable to very large datasets. In this paper, we propose a novel LS-based method, the best orthogonalized subset selection (BOSS) method, which performs BS upon an orthogonalized basis of ordered predictors and scales easily to large problem sizes. Another challenge in applying LS-based methods in practice is the selection rule to choose the optimal subset size k. Cross-validation (CV) requires fitting a procedure multiple times, and results in a selected k that is random across repeated application to the same dataset. Compared to CV, information criteria only require fitting a procedure once, but they require knowledge of the effective degrees of freedom for the fitting procedure, which is generally not available analytically for complex methods. Since BOSS uses orthogonalized predictors, we first explore a connection for orthogonal non-random predictors between BS and its Lagrangian formulation (i.e., minimization of the residual sum of squares plus the product of a regularization parameter and k), and based on this connection propose a heuristic degrees of freedom (hdf) for BOSS that can be estimated via an analytically-based expression. We show in both simulations and real data analysis that BOSS using a proposed Kullback-Leibler based information criterion AICc-hdf has the strongest performance of all of the LS-based methods considered and is competitive with regularization methods, with the computational effort of a single ordinary LS fit.