Label Errors in the Tobacco3482 Dataset

TL;DR

This paper identifies widespread label errors in the Tobacco3482 dataset, revealing that a significant portion of annotations are incorrect or ambiguous, which impacts model evaluation and highlights the need for cleaner datasets.

Contribution

The study provides a comprehensive manual review of Tobacco3482, quantifies label errors, and analyzes their effect on model performance, offering insights for dataset improvement.

Findings

11.7% of labels are incorrect or ambiguous

16.7% of samples have multiple valid labels

35% of model errors are due to label issues

Abstract

Tobacco3482 is a widely used document classification benchmark dataset. However, our manual inspection of the entire dataset uncovers widespread ontological issues, especially large amounts of annotation label problems in the dataset. We establish data label guidelines and find that 11.7% of the dataset is improperly annotated and should either have an unknown label or a corrected label, and 16.7% of samples in the dataset have multiple valid labels. We then analyze the mistakes of a top-performing model and find that 35% of the model's mistakes can be directly attributed to these label issues, highlighting the inherent problems with using a noisily labeled dataset as a benchmark. Supplementary material, including dataset annotations and code, is available at https://github.com/gordon-lim/tobacco3482-mistakes/.