The Efficiency Gap in Byte Modeling

TL;DR

This paper examines the computational costs of byte-level modeling and masked diffusion modeling in language models, revealing that byte modeling incurs higher scaling overhead for diffusion models due to context fragility.

Contribution

It provides a compute-matched scaling study comparing byte modeling and autoregressive approaches, highlighting the importance of structural biases for efficient scaling.

Findings

Byte modeling has worse scaling overhead for MDM than AR models.

Context fragility affects byte modeling efficiency in diffusion models.

Structural biases are crucial for viable scaling in byte-based, modality-agnostic models.

Abstract

Modern language models have historically relied on two dominant design choices: subword tokenization and autoregressive (AR) ordering. These design decisions bake in priors that dictate a model's learning. Recently, two alternative paradigms have challenged this: byte-level modeling, which bypasses static statistically-derived token vocabularies, and masked diffusion modeling (MDM), which conducts parallel, non-sequential generation. Their intersection represents a fully end-to-end modality-agnostic generative prototype; however, removing these structural priors incurs a significant computational cost. In this work, we investigate this cost through a compute-matched scaling study. Our results reveal that the performance penalty of byte modeling is not uniform; across scale, the scaling overhead of byte modeling is worse for MDM than for AR. We hypothesize that this disparity stems from context fragility: while AR's stable causal history allows models to naturally rediscover subword patterns, the MDM objective destroys the local contiguity required to efficiently resolve semantics from raw bytes. Our findings from controlled permutation experiments suggest that future modality-agnostic designs must incorporate alternative structural biases to maintain viable scaling trajectories in the byte regime.