From Data to Knowledge: Evaluating How Efficiently Language Models Learn Facts

TL;DR

This paper investigates how different language models learn and recall facts with varying frequencies in training data, revealing that models perform similarly on common facts but differ on rare ones, highlighting the importance of sample efficiency.

Contribution

It provides a comparative analysis of multiple models' ability to learn facts across frequency ranges, offering new insights into the impact of architecture and size on factual learning efficiency.

Findings

Models perform similarly on high-frequency facts

Significant differences emerge on low-frequency facts

Insights into architecture and size effects on factual learning

Abstract

Sample efficiency is a crucial property of language models with practical implications for training efficiency. In real-world text, information follows a long-tailed distribution. Yet, we expect models to learn and recall frequent and infrequent facts. Sample-efficient models are better equipped to handle this challenge of learning and retaining rare information without requiring excessive exposure. This study analyzes multiple models of varying architectures and sizes, all trained on the same pre-training data. By annotating relational facts with their frequencies in the training corpus, we examine how model performance varies with fact frequency. Our findings show that most models perform similarly on high-frequency facts but differ notably on low-frequency facts. This analysis provides new insights into the relationship between model architecture, size, and factual learning efficiency.