An Empirical Study of Memorization in NLP

TL;DR

This study empirically investigates memorization in NLP models across three tasks, confirming the long-tail theory and revealing that top-memorized instances are atypical and negatively correlated with class labels.

Contribution

It provides the first empirical verification of the long-tail memorization theory in NLP and introduces a novel attribution method to analyze memorization.

Findings

Top-memorized instances are often atypical.

Removing top-memorized instances significantly drops test accuracy.

Top-memorized features are negatively correlated with labels.

Abstract

A recent study by Feldman (2020) proposed a long-tail theory to explain the memorization behavior of deep learning models. However, memorization has not been empirically verified in the context of NLP, a gap addressed by this work. In this paper, we use three different NLP tasks to check if the long-tail theory holds. Our experiments demonstrate that top-ranked memorized training instances are likely atypical, and removing the top-memorized training instances leads to a more serious drop in test accuracy compared with removing training instances randomly. Furthermore, we develop an attribution method to better understand why a training instance is memorized. We empirically show that our memorization attribution method is faithful, and share our interesting finding that the top-memorized parts of a training instance tend to be features negatively correlated with the class label.