Robustness Auditing for Linear Regression: To Singularity and Beyond

TL;DR

This paper introduces an efficient algorithm to certify the robustness of linear regression models against sample removal, addressing limitations of previous methods and providing the first non-trivial certificates for high-dimensional econometrics datasets.

Contribution

We develop a novel, computationally efficient algorithm for certifying linear regression robustness to sample removal, applicable to large, high-dimensional datasets, with tight bounds under certain assumptions.

Findings

Algorithm successfully certifies robustness on datasets with hundreds of dimensions.

First non-trivial robustness certificates achieved for datasets of dimension 4 or higher.

Bounds are tight up to a 1 + o(1) factor under distributional assumptions.

Abstract

It has recently been discovered that the conclusions of many highly influential econometrics studies can be overturned by removing a very small fraction of their samples (often less than $0.5\%$). These conclusions are typically based on the results of one or more Ordinary Least Squares (OLS) regressions, raising the question: given a dataset, can we certify the robustness of an OLS fit on this dataset to the removal of a given number of samples? Brute-force techniques quickly break down even on small datasets. Existing approaches which go beyond brute force either can only find candidate small subsets to remove (but cannot certify their non-existence) [BGM20, KZC21], are computationally intractable beyond low dimensional settings [MR22], or require very strong assumptions on the data distribution and too many samples to give reasonable bounds in practice [BP21, FH23]. We present an efficient algorithm for certifying the robustness of linear regressions to removals of samples. We implement our algorithm and run it on several landmark econometrics datasets with hundreds of dimensions and tens of thousands of samples, giving the first non-trivial certificates of robustness to sample removal for datasets of dimension $4$ or greater. We prove that under distributional assumptions on a dataset, the bounds produced by our algorithm are tight up to a $1 + o(1)$ multiplicative factor.