Distributionally robust and generalizable inference

TL;DR

This paper reviews methods for assessing the stability and robustness of statistical inferences under distributional shifts, crucial for reliable analysis when data deviates from ideal i.i.d. assumptions.

Contribution

It introduces and discusses two approaches for quantifying distribution stability from a single dataset, addressing both worst-case and average robustness.

Findings

Sensitivity analysis identifies shifts threatening validity.

Random shift modeling assesses average robustness.

Unified confidence intervals incorporate multiple uncertainties.

Abstract

We discuss recently developed methods that quantify the stability and generalizability of statistical findings under distributional changes. In many practical problems, the data is not drawn i.i.d. from the target population. For example, unobserved sampling bias, batch effects, or unknown associations might inflate the variance compared to i.i.d. sampling. For reliable statistical inference, it is thus necessary to account for these types of variation. We discuss and review two methods that allow quantifying distribution stability based on a single dataset. The first method computes the sensitivity of a parameter under worst-case distributional perturbations to understand which types of shift pose a threat to external validity. The second method treats distributional shifts as random which allows assessing average robustness (instead of worst-case). Based on a stability analysis of multiple estimators on a single dataset, it integrates both sampling and distributional uncertainty into a single confidence interval.