LARP: Learner-Agnostic Robust Data Prefiltering

TL;DR

This paper introduces a formal framework for learner-agnostic robust data prefiltering (LARP) that aims to improve data quality for multiple downstream learners, balancing robustness and utility through theoretical analysis and experiments.

Contribution

It formalizes the LARP problem, analyzes its theoretical hardness, and evaluates the utility trade-offs of learner-agnostic versus learner-specific prefiltering methods.

Findings

Prefiltering for multiple learners can lead to utility loss compared to learner-specific approaches.

Theoretical hardness results highlight challenges in universal prefiltering.

Experimental results show statistically significant utility reduction with learner-agnostic prefiltering.

Abstract

The widespread availability of large public datasets is a key factor behind the recent successes of statistical inference and machine learning methods. However, these datasets often contain some low-quality or contaminated data, to which many learning procedures are sensitive. Therefore, the question of whether and how public datasets should be prefiltered to facilitate accurate downstream learning arises. On a technical level this requires the construction of principled data prefiltering methods which are learner-agnostic robust, in the sense of provably protecting a set of pre-specified downstream learners from corrupted data. In this work, we formalize the problem of Learner-Agnostic Robust data Prefiltering (LARP), which aims at finding prefiltering procedures that minimize a worst-case loss over a pre-specified set of learners. We first instantiate our framework in the context of scalar mean estimation with Huber estimators under the Huber data contamination model. We provide a hardness result on a specific problem instance and analyze several natural prefiltering procedures. Our theoretical results indicate that performing LARP on a heterogeneous set of learners leads to some loss in model performance compared to the alternative of prefiltering data for each learner/use-case individually. We explore the resulting utility loss and its dependence on the problem parameters via extensive experiments on real-world image and tabular data, observing statistically significant reduction in utility. Finally, we model the trade-off between the utility drop and the cost of repeated (learner-specific) prefiltering within a game-theoretic framework and showcase benefits of LARP for large datasets.