Memorization vs. Generalization: Quantifying Data Leakage in NLP Performance Evaluation

TL;DR

This paper investigates how data leakage in NLP datasets inflates performance metrics, blurring the line between memorization and true generalization, and assesses its impact on evaluating NLP models.

Contribution

It quantifies data leakage in popular NLP datasets and analyzes its effect on model evaluation, highlighting the need for more reliable benchmarking.

Findings

Leakage inflates performance metrics significantly.

Models tend to memorize leaked data rather than generalize.

Current datasets may not accurately reflect real-world performance.

Abstract

Public datasets are often used to evaluate the efficacy and generalizability of state-of-the-art methods for many tasks in natural language processing (NLP). However, the presence of overlap between the train and test datasets can lead to inflated results, inadvertently evaluating the model's ability to memorize and interpreting it as the ability to generalize. In addition, such data sets may not provide an effective indicator of the performance of these methods in real world scenarios. We identify leakage of training data into test data on several publicly available datasets used to evaluate NLP tasks, including named entity recognition and relation extraction, and study them to assess the impact of that leakage on the model's ability to memorize versus generalize.