How to Sift Out a Clean Data Subset in the Presence of Data Poisoning?

TL;DR

This paper investigates the difficulty of identifying clean data in poisoned datasets, analyzes existing defenses' vulnerabilities, and introduces Meta-Sift, a method that effectively isolates clean data for robust machine learning.

Contribution

The paper reveals the limitations of current automated and human methods for detecting clean data in poisoned datasets and proposes Meta-Sift, a bilevel optimization approach that achieves perfect precision in identifying clean data.

Findings

Existing defenses fail with less than 1% poisoned data in the base set.

Automated tools and human inspection often perform worse than random chance.

Meta-Sift achieves 100% precision in extracting clean data across various attacks.

Abstract

Given the volume of data needed to train modern machine learning models, external suppliers are increasingly used. However, incorporating external data poses data poisoning risks, wherein attackers manipulate their data to degrade model utility or integrity. Most poisoning defenses presume access to a set of clean data (or base set). While this assumption has been taken for granted, given the fast-growing research on stealthy poisoning attacks, a question arises: can defenders really identify a clean subset within a contaminated dataset to support defenses? This paper starts by examining the impact of poisoned samples on defenses when they are mistakenly mixed into the base set. We analyze five defenses and find that their performance deteriorates dramatically with less than 1% poisoned points in the base set. These findings suggest that sifting out a base set with high precision is key to these defenses' performance. Motivated by these observations, we study how precise existing automated tools and human inspection are at identifying clean data in the presence of data poisoning. Unfortunately, neither effort achieves the precision needed. Worse yet, many of the outcomes are worse than random selection. In addition to uncovering the challenge, we propose a practical countermeasure, Meta-Sift. Our method is based on the insight that existing attacks' poisoned samples shifts from clean data distributions. Hence, training on the clean portion of a dataset and testing on the corrupted portion will result in high prediction loss. Leveraging the insight, we formulate a bilevel optimization to identify clean data and further introduce a suite of techniques to improve efficiency and precision. Our evaluation shows that Meta-Sift can sift a clean base set with 100% precision under a wide range of poisoning attacks. The selected base set is large enough to give rise to successful defenses.