Rethinking Distribution Shifts: Empirical Analysis and Inductive Modeling for Tabular Data

TL;DR

This paper empirically analyzes distribution shifts in tabular data, revealing that $Y|X$-shifts are common and that robust algorithms often underperform due to implementation details, advocating for data-driven, inductive modeling approaches.

Contribution

It provides an empirical testbed for distribution shifts in tabular data and highlights the importance of implementation choices over theoretical robustness assumptions.

Findings

$Y|X$-shifts are most common in the testbed.

Robust algorithms do not outperform vanilla methods.

Implementation details significantly impact performance.

Abstract

Different distribution shifts require different interventions, and algorithms must be grounded in the specific shifts they address. However, methodological development for robust algorithms typically relies on structural assumptions that lack empirical validation. Advocating for an empirically grounded data-driven approach to algorithm development, we build an empirical testbed comprising natural shifts across 8 tabular datasets, 172 distribution pairs over 45 methods and 90,000 method configurations encompassing empirical risk minimization and distributionally robust optimization (DRO) methods. We find $Y|X$-shifts are most prevalent in our testbed, in stark contrast to the heavy focus on $X$ (covariate)-shifts in the ML literature, and that the performance of robust algorithms is no better than that of vanilla methods. To understand why, we conduct an in-depth empirical analysis of DRO methods and find that underlooked implementation details -- such as the choice of underlying model class (e.g., LightGBM) and hyperparameter selection -- have a bigger impact on performance than the ambiguity set or its radius. We illustrate via case studies how a data-driven, inductive understanding of distribution shifts can provide a new approach to algorithm development.