PreCog: Exploring the Relation between Memorization and Performance in Pre-trained Language Models

TL;DR

This paper introduces PreCog, a measure to evaluate memorization in BERT, revealing that highly memorized examples tend to be classified better, indicating memorization's role in BERT's success.

Contribution

The paper proposes PreCog, a novel metric for assessing memorization in pre-trained language models, and analyzes its correlation with downstream task performance.

Findings

Highly memorized examples are better classified by BERT.

Memorization correlates positively with BERT's performance.

Memorization is an essential factor for BERT's success.

Abstract

Pre-trained Language Models such as BERT are impressive machines with the ability to memorize, possibly generalized learning examples. We present here a small, focused contribution to the analysis of the interplay between memorization and performance of BERT in downstream tasks. We propose PreCog, a measure for evaluating memorization from pre-training, and we analyze its correlation with the BERT's performance. Our experiments show that highly memorized examples are better classified, suggesting memorization is an essential key to success for BERT.