Robustness of OLS to sample removals: Theoretical analysis and implications

TL;DR

This paper provides a theoretical analysis of the robustness of ordinary least squares (OLS) regression to the removal of subsets of training samples, identifying conditions under which OLS remains stable and highlighting factors affecting its robustness.

Contribution

It offers the first theoretical bounds on OLS robustness to sample removals, clarifies when OLS is non-robust, and connects these findings to empirical observations in econometrics.

Findings

OLS is robust to small sample removals with high probability.

Robustness limit is up to ${k extless \sqrt{np}/\log n}$ sample removals.

Heavy-tailed responses and correlated samples reduce robustness.

Abstract

For learned models to be trustworthy, it is essential to verify their robustness to perturbations in the training data. Classical approaches involve uncertainty quantification via confidence intervals and bootstrap methods. In contrast, recent work proposes a more stringent form of robustness: stability to the removal of any subset of $k$ samples from the training set. In this paper, we present a theoretical study of this criterion for ordinary least squares (OLS). Our contributions are as follows: (1) Given $n$ i.i.d. training samples from a general misspecified model, we prove that with high probability, OLS is robust to the removal of any $k \ll n $ samples. (2) For data of dimension $p$, OLS can withstand up to ${k\ll \sqrt{np}/\log n}$ sample removals while remaining robust and achieving the same error rate as OLS applied to the full dataset. Conversely, if $k$ is proportional to $n$, OLS is provably non-robust. (3) We revisit prior analyses that found several econometric datasets to be highly non-robust to sample removals. While this appears to contradict our results in (1), we demonstrate that the sensitivity is due to either heavy-tailed responses or correlated samples. Empirically, this sensitivity is considerably attenuated by classical robust methods, such as linear regression with a Huber loss.