Bolmo: Byteifying the Next Generation of Language Models

TL;DR

Bolmo introduces a family of byte-level language models that match subword-based models in performance, enabling fine-grained text understanding and efficient inference across various domains, including scientific data.

Contribution

The paper presents Bolmo, a novel approach to convert existing subword models into byte-level models with minimal training, achieving competitive performance and reasoning capabilities.

Findings

Outperforms previous byte-level models

Excels on character-level reasoning tasks

Remains competitive on standard benchmarks

Abstract

Recent advances in generative AI have been largely driven by large language models (LLMs), deep neural networks that operate over discrete units called tokens. To represent text, the vast majority of LLMs use words or word fragments as the tokens, known as subword tokenization. Subword tokenization obscures fine-grained information, which is problematic, especially for scientific data - such as computer code or biological sequences - where meaning depends on the individual characters. Models that instead operate directly on the byte encoding of text avoid these limitations, but until now they have lagged behind subword-based models in performance. Here we introduce Bolmo, a family of fully open byte-level LLMs that approach the capabilities of subword-based systems. Using a two-stage conversion procedure, we transform existing subword-based models into byte-level models with minimal additional training. The resulting models outperform prior byte-level approaches and excel on character-level reasoning tasks, while remaining competitive across standard benchmarks. By efficiently processing byte-level information, these models achieve practical inference speeds and can be adapted at low cost using the existing ecosystem around the source LLM. Our results remove a long-standing performance barrier to end-to-end byte-level language modeling, demonstrating that models operating on raw text encodings can scale competitively while offering advantages in domains requiring fine-grained textual understanding.