Robust subset selection

TL;DR

This paper introduces a robust version of best subset selection for sparse regression that resists outliers in data, combining combinatorial optimization with robustness guarantees and demonstrating superior performance on contaminated datasets.

Contribution

It proposes a novel robust subset selection method that generalizes traditional subset selection to handle outliers in both predictors and responses, with formal robustness analysis.

Findings

Robust subsets outperform traditional best subsets under data contamination.

The method achieves high finite-sample breakdown point.

Experimental results show competitive performance with popular robust estimators.

Abstract

The best subset selection (or "best subsets") estimator is a classic tool for sparse regression, and developments in mathematical optimization over the past decade have made it more computationally tractable than ever. Notwithstanding its desirable statistical properties, the best subsets estimator is susceptible to outliers and can break down in the presence of a single contaminated data point. To address this issue, a robust adaption of best subsets is proposed that is highly resistant to contamination in both the response and the predictors. The adapted estimator generalizes the notion of subset selection to both predictors and observations, thereby achieving robustness in addition to sparsity. This procedure, referred to as "robust subset selection" (or "robust subsets"), is defined by a combinatorial optimization problem for which modern discrete optimization methods are applied. The robustness of the estimator in terms of the finite-sample breakdown point of its objective value is formally established. In support of this result, experiments on synthetic and real data are reported that demonstrate the superiority of robust subsets over best subsets in the presence of contamination. Importantly, robust subsets fares competitively across several metrics compared with popular robust adaptions of continuous shrinkage estimators.