Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws

TL;DR

This paper investigates how language models store factual knowledge, establishing that they can store approximately 2 bits per parameter, and explores factors affecting this capacity, including architecture, training, and data signals.

Contribution

It introduces a method to estimate knowledge capacity in language models and provides new insights into how architecture and training influence knowledge storage.

Findings

Language models store about 2 bits of knowledge per parameter.

A 7B model can store more knowledge than entire Wikipedia and textbooks.

Prepending domain names to training data enhances knowledge capacity.

Abstract

Scaling laws describe the relationship between the size of language models and their capabilities. Unlike prior studies that evaluate a model's capability via loss or benchmarks, we estimate the number of knowledge bits a model stores. We focus on factual knowledge represented as tuples, such as (USA, capital, Washington D.C.) from a Wikipedia page. Through multiple controlled datasets, we establish that language models can and only can store 2 bits of knowledge per parameter, even when quantized to int8, and such knowledge can be flexibly extracted for downstream applications. Consequently, a 7B model can store 14B bits of knowledge, surpassing the English Wikipedia and textbooks combined based on our estimation. More broadly, we present 12 results on how (1) training duration, (2) model architecture, (3) quantization, (4) sparsity constraints such as MoE, and (5) data signal-to-noise ratio affect a model's knowledge storage capacity. Notable insights include: * The GPT-2 architecture, with rotary embedding, matches or even surpasses LLaMA/Mistral architectures in knowledge storage, particularly over shorter training durations. This arises because LLaMA/Mistral uses GatedMLP, which is less stable and harder to train. * Prepending training data with domain names (e.g., wikipedia.org) significantly increases a model's knowledge capacity. Language models can autonomously identify and prioritize domains rich in knowledge, optimizing their storage capacity.