Know Your Limits: Entropy Estimation Modeling for Compression and Generalization

TL;DR

This paper introduces encoder-augmented causal decoder models that efficiently estimate language entropy, leading to improved compression and better generalization by approaching the theoretical entropy limits.

Contribution

It proposes novel encoder-augmented causal decoder architectures for efficient entropy estimation and demonstrates their advantages in compression and generalization over traditional models.

Findings

Encoder-augmented models achieve higher compression than causal transformers.

Models trained near estimated entropy bounds generalize better.

Entropy-aware training improves language model performance.

Abstract

Language prediction is constrained by informational entropy intrinsic to language, such that there exists a limit to how accurate any language model can become and equivalently a lower bound to language compression. The most efficient language compression algorithms today are causal (next token prediction) large language models, but the use of these models to form accurate estimates of language entropy is currently computationally infeasible. We introduce encoder-augmented causal decoder model architectures that exhibit superior training efficiency characteristics and achieve higher compression than causal transformers even when trained on modest hardware. We demonstrate how entropy estimates can be obtained on a per-token basis, and show that the generalization of models trained to approach the entropy of their training data necessarily exceeds the generalization of models trained to minimize loss beyond this value. We show empirically that causal models trained to approach but not exceed estimated per-token entropies exhibit greater generalization than models trained without taking entropy into account.