Dynamic Chunking for End-to-End Hierarchical Sequence Modeling

TL;DR

This paper introduces a dynamic chunking mechanism within hierarchical models that learns content-dependent segmentation, enabling fully end-to-end language modeling that outperforms traditional token-based models, especially in languages with weak tokenization.

Contribution

The paper presents a novel hierarchical network (H-Net) with dynamic chunking that replaces tokenization, enabling end-to-end training and improved performance across languages and modalities.

Findings

H-Net with one hierarchy stage outperforms BPE-based Transformers.

Multiple hierarchy stages improve abstraction and scaling.

Significant data efficiency gains in Chinese, code, and DNA sequences.

Abstract

Major progress on language models (LMs) in recent years has largely resulted from moving away from specialized models designed for specific tasks, to general models based on powerful architectures (e.g. the Transformer) that learn everything from raw data. Despite this trend, pre-processing steps such as tokenization remain a barrier to true end-to-end foundation models. We introduce a collection of new techniques that enable a dynamic chunking mechanism which automatically learns content- and context- dependent segmentation strategies learned jointly with the rest of the model. Incorporating this into an explicit hierarchical network (H-Net) allows replacing the (implicitly hierarchical) tokenization-LM-detokenization pipeline with a single model learned fully end-to-end. When compute- and data- matched, an H-Net with one stage of hierarchy operating at the byte level outperforms a strong Transformer language model operating over BPE tokens. Iterating the hierarchy to multiple stages further increases its performance by modeling multiple levels of abstraction, demonstrating significantly better scaling with data and matching the token-based Transformer of twice its size. H-Nets pretrained on English show significantly increased character-level robustness, and qualitatively learn meaningful data-dependent chunking strategies without any heuristics or explicit supervision. Finally, the H-Net's improvement over tokenized pipelines is further increased in languages and modalities with weaker tokenization heuristics, such as Chinese and code, or DNA sequences (nearly 4x improvement in data efficiency over baselines), showing the potential of true end-to-end models that learn and scale better from unprocessed data.