Robust Grouped Variable Selection Using Distributionally Robust Optimization

TL;DR

This paper introduces a distributionally robust optimization framework for grouped variable selection that enhances model interpretability, sparsity, and prediction accuracy, especially under data perturbations and outliers.

Contribution

It develops a Wasserstein-based DRO model for grouped variable selection, linking robustness with regularization, and proposes a spectral clustering method for unknown group structures.

Findings

Improves prediction and estimation in presence of outliers

Provides probabilistic bounds on out-of-sample loss and bias

Demonstrates effectiveness on synthetic and real datasets

Abstract

We propose a Distributionally Robust Optimization (DRO) formulation with a Wasserstein-based uncertainty set for selecting grouped variables under perturbations on the data for both linear regression and classification problems. The resulting model offers robustness explanations for Grouped Least Absolute Shrinkage and Selection Operator (GLASSO) algorithms and highlights the connection between robustness and regularization. We prove probabilistic bounds on the out-of-sample loss and the estimation bias, and establish the grouping effect of our estimator, showing that coefficients in the same group converge to the same value as the sample correlation between covariates approaches 1. Based on this result, we propose to use the spectral clustering algorithm with the Gaussian similarity function to perform grouping on the predictors, which makes our approach applicable without knowing the grouping structure a priori. We compare our approach to an array of alternatives and provide extensive numerical results on both synthetic data and a real large dataset of surgery-related medical records, showing that our formulation produces an interpretable and parsimonious model that encourages sparsity at a group level and is able to achieve better prediction and estimation performance in the presence of outliers.