CharBench: Evaluating the Role of Tokenization in Character-Level Tasks

TL;DR

CharBench is a large benchmark designed to evaluate how tokenization affects character-level reasoning tasks in language models, revealing that current models struggle significantly with these tasks.

Contribution

The paper introduces CharBench, a comprehensive and larger benchmark for character-level tasks, and provides detailed analysis of tokenization's impact on model performance.

Findings

Modern LLMs achieve around 43.6% accuracy on CharBench tasks.

Tokenization properties are weakly correlated with counting task accuracy.

Longer tokens obscure character position information, negatively affecting intra-word positional tasks.

Abstract

Tasks that require character-level reasoning, such as counting or locating characters within words, remain challenging for contemporary language models. A common conjecture is that language models' reliance on subword units, rather than characters, contributes to their struggles with character-level tasks, yet recent studies offer conflicting conclusions about the role of tokenization, leaving its impact unclear. To address this gap, we introduce CharBench, a comprehensive benchmark of character-level tasks that is two orders of magnitude larger than existing alternatives. We evaluate a diverse range of leading open-weight and proprietary models on CharBench and find that it presents a significant challenge to modern LLMs, with an average accuracy of 43.6% and 32.3% on some tasks. We present an in-depth analysis of how intrinsic properties of words and their segmentations into tokens correspond to model performance. For counting tasks, we find that tokenization properties are weakly correlated with correctness, while the length of the queried word and the actual character count play a more significant part. In contrast, for tasks requiring intra-word positional understanding, performance is negatively correlated with the length of the token containing the queried character, suggesting that longer tokens obscure character position information for LLMs. We encourage future work to build on the benchmark and evaluation methodology introduced here as tools for improving model performance on such tasks.