Information Capacity: Evaluating the Efficiency of Large Language Models via Text Compression

TL;DR

This paper introduces 'information capacity', a new metric for evaluating large language models' inference efficiency based on text compression, accounting for tokenizer efficiency and correlating with model performance and linguistic biases.

Contribution

The paper proposes a novel efficiency metric for LLMs that incorporates tokenizer efficiency and correlates with model performance, aiding future model scaling and optimization.

Findings

Information capacity is consistent across models of different sizes within a series.

Strong linguistic biases are observed in mainstream LLMs.

Performance can be accurately predicted from information capacity.

Abstract

Recent years have witnessed the rapid advancements of large language models (LLMs) and their expanding applications, leading to soaring demands for computational resources. The widespread adoption of test-time scaling further intensifies the tension between model capability and resource consumption. However, a rigorous metric that accurately reflects an LLM's inference efficiency across diverse tokenizers, parameter counts, and model architectures remains absent. Motivated by the correlation between compression and intelligence, we introduce information capacity, a measure of model efficiency based on text compression performance relative to computational complexity. A distinctive feature of information capacity is its incorporation of tokenizer efficiency, which affects inference costs but is often neglected in LLM evaluations. We assess the information capacity of 56 open-source models and observe a consistent information capacity among different-sized models within a series. Experiments on five heterogeneous datasets reveal strong linguistic biases in mainstream LLMs. Empirical results verify the accuracy of performance prediction across model sizes based on information capacity and show the correlation between information capacity and benchmark scores. This metric can be used to quantify improvements in inference efficiency and provide insights into better scaling performance for future LLM development.