Provable Training Set Debugging for Linear Regression

TL;DR

This paper develops a provable method for identifying and correcting contaminated data points in linear regression models, combining theoretical guarantees with practical algorithms and game-theoretic insights.

Contribution

It introduces a Lasso-based debugging algorithm with theoretical guarantees, analyzes a game-theoretic model of data contamination, and demonstrates practical effectiveness through case studies.

Findings

The Lasso-based method reliably identifies buggy points under certain conditions.

A theoretical condition allows the bug generator to fool the debugger, but it is unlikely in practice.

Empirical results show successful debugging with natural data augmentation strategies.

Abstract

We investigate problems in penalized $M$-estimation, inspired by applications in machine learning debugging. Data are collected from two pools, one containing data with possibly contaminated labels, and the other which is known to contain only cleanly labeled points. We first formulate a general statistical algorithm for identifying buggy points and provide rigorous theoretical guarantees under the assumption that the data follow a linear model. We then present two case studies to illustrate the results of our general theory and the dependence of our estimator on clean versus buggy points. We further propose an algorithm for tuning parameter selection of our Lasso-based algorithm and provide corresponding theoretical guarantees. Finally, we consider a two-person "game" played between a bug generator and a debugger, where the debugger can augment the contaminated data set with cleanly labeled versions of points in the original data pool. We establish a theoretical result showing a sufficient condition under which the bug generator can always fool the debugger. Nonetheless, we provide empirical results showing that such a situation may not occur in practice, making it possible for natural augmentation strategies combined with our Lasso debugging algorithm to succeed.