Does Visual Rendering Bypass Tokenization? Investigating Script-Tokenizer Misalignment in Pixel-Based Language Models

TL;DR

This paper investigates whether pixel-based language models truly bypass tokenization issues, finding that reintegrating tokenizers reintroduces misalignment problems, especially affecting low-resource languages with unique scripts.

Contribution

The study reveals that visual rendering does not eliminate tokenization constraints and highlights the importance of tokenizer design for low-resource language modeling.

Findings

Reintegrating tokenizers reintroduces misalignment issues.

Custom tokenizers outperform standard ones by up to 30.15 chrF++.

Visual rendering alone does not solve tokenization barriers.

Abstract

While pixel-based language modeling aims to bypass the sub-word tokenization bottleneck by rendering text as images, recent multimodal variants such as DualGPT reintroduce text tokenizers to improve autoregressive performance. We investigate a fundamental question, does visual rendering truly decouple a model from tokenization constraints? Focusing on four Indonesian low-resource local languages that have their own non-Latin scripts (i.e., Javanese, Balinese, Sundanese, and Lampungnese), we evaluate the impact of script-tokenizer alignment within the DualGPT architecture. Our results show that, despite visual rendering, reintegrating a text tokenizer into the architecture reintroduces the same issue that pixel-based language modeling aims to resolve, which is the tokenizer misalignment problem. Despite having lower OOV and fertility rates, we show that the Llama 2 tokenizer performs significantly worse than a custom tokenizer, with improvements of up to 30.15 chrF++. Our findings serve as a warning for future multimodal variants, as text tokenizers remain a significant barrier to equitable models.