Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisition

TL;DR

This paper introduces the concept of knowledge entropy to measure how language models' engagement with diverse memory sources declines during pretraining, impairing their ability to acquire and retain new knowledge.

Contribution

The study defines knowledge entropy, demonstrates its decline during pretraining, and shows that increasing inactive memory sources can improve knowledge acquisition.

Findings

Knowledge entropy decreases as pretraining progresses.

Lower knowledge entropy correlates with reduced knowledge retention.

Activating inactive memory sources enhances knowledge acquisition.

Abstract

In this work, we investigate how a model's tendency to broadly integrate its parametric knowledge evolves throughout pretraining, and how this behavior affects overall performance, particularly in terms of knowledge acquisition and forgetting. We introduce the concept of knowledge entropy, which quantifies the range of memory sources the model engages with; high knowledge entropy indicates that the model utilizes a wide range of memory sources, while low knowledge entropy suggests reliance on specific sources with greater certainty. Our analysis reveals a consistent decline in knowledge entropy as pretraining advances. We also find that the decline is closely associated with a reduction in the model's ability to acquire and retain knowledge, leading us to conclude that diminishing knowledge entropy (smaller number of active memory sources) impairs the model's knowledge acquisition and retention capabilities. We find further support for this by demonstrating that increasing the activity of inactive memory sources enhances the model's capacity for knowledge acquisition and retention.