Tiny Transformers Excel at Sentence Compression

TL;DR

This paper demonstrates that tiny 1-3 layer transformer models can effectively encode and decode English sentences into extremely compact token representations, suggesting potential for optimizing language models by using larger text fragments.

Contribution

The study shows that small transformers can compress and reconstruct English sentences into minimal token sizes, challenging assumptions about model size and tokenization.

Findings

Tiny transformers encode sentences into as little as 3 KB tokens

Small models can accurately reconstruct valid English sentences

Implications for optimizing language models with larger text fragments

Abstract

It is staggering that words of the English language, which are on average represented by 5--6 bytes of ASCII, require as much as 24 kilobytes when served to large language models. We show that there is room for more information in every token embedding. We demonstrate that 1--3-layer transformers are capable of encoding and subsequently decoding standard English sentences into as little as a single 3-kilobyte token. Our work implies that even small networks can learn to construct valid English sentences and suggests the possibility of optimising large language models by moving from sub-word token embeddings towards larger fragments of text.