NoisywikiHow: A Benchmark for Learning with Real-world Noisy Labels in Natural Language Processing

TL;DR

This paper introduces NoisywikiHow, a large NLP benchmark with realistic, heterogeneous label noise inspired by human errors, enabling systematic evaluation of learning with noisy labels in real-world scenarios.

Contribution

It presents the first large-scale NLP benchmark with instance-dependent noise mimicking real-world errors, supporting controlled experiments for better LNL method evaluation.

Findings

LNL methods show varied performance across noise levels.

Realistic noise impacts model generalization more than synthetic noise.

Benchmark enables comprehensive evaluation of noise-robust learning techniques.

Abstract

Large-scale datasets in the real world inevitably involve label noise. Deep models can gradually overfit noisy labels and thus degrade model generalization. To mitigate the effects of label noise, learning with noisy labels (LNL) methods are designed to achieve better generalization performance. Due to the lack of suitable datasets, previous studies have frequently employed synthetic label noise to mimic real-world label noise. However, synthetic noise is not instance-dependent, making this approximation not always effective in practice. Recent research has proposed benchmarks for learning with real-world noisy labels. However, the noise sources within may be single or fuzzy, making benchmarks different from data with heterogeneous label noises in the real world. To tackle these issues, we contribute NoisywikiHow, the largest NLP benchmark built with minimal supervision. Specifically, inspired by human cognition, we explicitly construct multiple sources of label noise to imitate human errors throughout the annotation, replicating real-world noise, whose corruption is affected by both ground-truth labels and instances. Moreover, we provide a variety of noise levels to support controlled experiments on noisy data, enabling us to evaluate LNL methods systematically and comprehensively. After that, we conduct extensive multi-dimensional experiments on a broad range of LNL methods, obtaining new and intriguing findings.