Robust Variable and Interaction Selection for Logistic Regression and Multiple Index Models

TL;DR

This paper introduces SODA, a robust stepwise method for variable and interaction selection in logistic regression and multiple index models, effective in high-dimensional, non-Gaussian data.

Contribution

SODA is a novel, robust variable selection method that handles high-dimensional data without normality assumptions, extending to multiple index models.

Findings

SODA outperforms existing methods in high-dimensional QDA variable selection.

SODA is robust to non-Gaussian predictor distributions.

Theoretical guarantees confirm SODA's variable-selection consistency.

Abstract

We propose Stepwise cOnditional likelihood variable selection for Discriminant Analysis (SODA) to detect both main and quadratic interaction effects in logistic regression and quadratic discriminant analysis (QDA) models. In the forward stage, SODA adds in important predictors evaluated based on their overall contributions, whereas in the backward stage SODA removes unimportant terms so as to optimize the extended Bayesian Information Criterion (EBIC). Compared with existing methods on QDA variable selections, SODA can deal with high-dimensional data with the number of predictors much larger than the sample size and does not require the joint normality assumption on predictors, leading to much enhanced robustness. We further extend SODA to conduct variable selection and model fitting for multiple index models. Compared with existing variable selection methods based on the Sliced Inverse Regression (SIR) (Li 1991), SODA requires neither the linearity nor the constant variance condition and is much more robust. Our theoretical analyses establish the variable-selection consistency of SODA under high-dimensional settings, and our simulation studies as well as real-data applications demonstrate superior performances of SODA in dealing with non-Gaussian design matrices in both classification problems and multiple index models.