A Comparative Analysis of Influence Signals for Data Debugging

TL;DR

This paper evaluates various influence-based signals for data debugging in machine learning, revealing their strengths and limitations in detecting mislabeled and anomalous samples across different data types and models.

Contribution

It provides a comprehensive experimental comparison of influence signals, highlighting their effectiveness and shortcomings in identifying data glitches during training.

Findings

Self-Influence effectively detects mislabeled samples.

Existing signals fail to detect anomalies.

Training dynamics are crucial for influence signal effectiveness.

Abstract

Improving the quality of training samples is crucial for improving the reliability and performance of ML models. In this paper, we conduct a comparative evaluation of influence-based signals for debugging training data. These signals can potentially identify both mislabeled and anomalous samples from a potentially noisy training set as we build the models and hence alleviate the need for dedicated glitch detectors. Although several influence-based signals (e.g., Self-Influence, Average Absolute Influence, Marginal Influence, GD-class) have been recently proposed in the literature, there are no experimental studies for assessing their power in detecting different glitch types (e.g., mislabeled and anomalous samples) under a common influence estimator (e.g., TraceIn) for different data modalities (image and tabular), and deep learning models (trained from scratch or foundation). Through extensive experiments, we show that signals like Self-Influence effectively detect mislabeled samples, but none of the existing signals can detect anomalies. Existing signals do not take into account the training dynamics, i.e., how the samples' influence on the model changes during training, while some signals fall into influence cancellation effects, i.e., influence score is zero due to unsigned scores accumulation, resulting in misleading influence attribution.