# Make Every Example Count: On the Stability and Utility of Self-Influence for Learning from Noisy NLP Datasets

## TL;DR

This paper evaluates the effectiveness of self-influence scores in identifying noisy data across NLP tasks, aiming to improve data quality and downstream performance without task-specific filtering rules.

## Contribution

It analyzes the stability and utility of self-influence scores for data cleaning in NLP, demonstrating their potential to enhance model performance across multiple tasks.

## Key findings

- Self-influence scores can identify outliers in NLP datasets.
- Self-influence based cleaning improves downstream task performance.
- Task-agnostic self-influence methods are effective across diverse NLP tasks.

## Abstract

Increasingly larger datasets have become a standard ingredient to advancing the state-of-the-art in NLP. However, data quality might have already become the bottleneck to unlock further gains. Given the diversity and the sizes of modern datasets, standard data filtering is not straight-forward to apply, because of the multifacetedness of the harmful data and elusiveness of filtering rules that would generalize across multiple tasks. We study the fitness of task-agnostic self-influence scores of training examples for data cleaning, analyze their efficacy in capturing naturally occurring outliers, and investigate to what extent self-influence based data cleaning can improve downstream performance in machine translation, question answering and text classification, building up on recent approaches to self-influence calculation and automated curriculum learning.