SOMBRERO: Measuring and Steering Boundary Placement in End-to-End Hierarchical Sequence Models

TL;DR

This paper introduces SOMBRERO, a method for quantitatively assessing and guiding boundary placement in hierarchical sequence models, improving efficiency and alignment with predictive difficulty across diverse data types.

Contribution

We propose a new boundary quality metric and a steering method that enhances boundary placement in hierarchical models, improving efficiency and alignment with challenging content.

Findings

Improved accuracy-efficiency trade-off on 1B scale datasets.

Boundaries more consistently align with high surprisal positions.

Enhanced model performance across multiple languages and content types.

Abstract

Hierarchical sequence models replace fixed tokenization with learned segmentations that compress long byte sequences for efficient autoregressive modeling. While recent end-to-end methods can learn meaningful boundaries from the language-modeling objective alone, it remains difficult to quantitatively assess and systematically steer where compute is spent. We introduce a router-agnostic metric of boundary quality, boundary enrichment B, which measures how strongly chunk starts concentrate on positions with high next-byte surprisal. Guided by this metric, we propose Sombrero, which steers boundary placement toward predictive difficulty via a confidence-alignment boundary loss and stabilizes boundary learning by applying confidence-weighted smoothing at the input level rather than on realized chunks. On 1B scale, across UTF-8 corpora covering English and German text as well as code and mathematical content, Sombrero improves the accuracy-efficiency trade-off and yields boundaries that more consistently align compute with hard-to-predict positions.