Best Subset Binary Prediction

TL;DR

This paper introduces a best subset variable selection method for binary outcome prediction using a mixed integer optimization approach, providing theoretical risk bounds and demonstrating practical effectiveness through simulations and real data.

Contribution

It reformulates the best subset selection for binary prediction as a mixed integer optimization problem, enabling exact or approximate solutions with theoretical guarantees.

Findings

Risk bounds are minimax rate-optimal when the number of selected variables is fixed.

The method can handle high-dimensional covariates exceeding sample size.

Empirical results show the approach's practical usefulness in real-world data.

Abstract

We consider a variable selection problem for the prediction of binary outcomes. We study the best subset selection procedure by which the covariates are chosen by maximizing Manski (1975, 1985)'s maximum score objective function subject to a constraint on the maximal number of selected variables. We show that this procedure can be equivalently reformulated as solving a mixed integer optimization problem, which enables computation of the exact or an approximate solution with a definite approximation error bound. In terms of theoretical results, we obtain non-asymptotic upper and lower risk bounds when the dimension of potential covariates is possibly much larger than the sample size. Our upper and lower risk bounds are minimax rate-optimal when the maximal number of selected variables is fixed and does not increase with the sample size. We illustrate usefulness of the best subset binary prediction approach via Monte Carlo simulations and an empirical application of the work-trip transportation mode choice.