Beyond Scaling: Measuring and Predicting the Upper Bound of Knowledge Retention in Language Model Pre-Training

TL;DR

This paper introduces Size-dependent Mutual Information (SMI), a method to predict a language model's knowledge retention capacity from pre-training signals, revealing an intrinsic upper bound and guiding future model scaling and augmentation strategies.

Contribution

The work presents SMI, a novel information-theoretic predictor for estimating knowledge retention in language models before training, validated across large-scale datasets and models.

Findings

SMI outperforms baselines in predicting QA accuracy

Diminishing returns from scaling data and model size

Evidence of an intrinsic upper bound on knowledge retention

Abstract

The GPT-4 technical report suggests that downstream performance can be predicted from pre-training signals, but offers little methodological detail on how to quantify this. This work address this gap by modeling knowledge retention, the capacity of a pre-trained language model to memorize factual information from its corpus, and introduce a principled method to estimate it prior to training. We propose Size-dependent Mutual Information (SMI), an information-theoretic predictor that integrates knowledge frequency, knowledge specificity, and model size to forecast closed-book question answering (QA) accuracy. SMI is validated through large-scale document retrieval over the disclosed pre-training corpora of 21 public and 3 custom models, combined with a robust multi-template QA evaluation. Experiments show that SMI significantly outperforms repetition-based baselines and achieves $R^2$ > 0.7 in predicting QA accuracy for models above 1B parameters, without additional training. The analysis further reveals diminishing returns from scaling data and model size and provides evidence for an intrinsic upper bound on knowledge retention achievable by pre-training alone, motivating retrieval and other augmentation strategies. The dataset and code are available at https://github.com/yuhui1038/SMI.