Egalitarian Language Representation in Language Models: It All Begins with Tokenizers

TL;DR

This paper investigates how tokenization affects language representation fairness in language models, especially for complex scripts, and introduces Grapheme Pair Encoding (GPE) to improve tokenization for these languages.

Contribution

It highlights the importance of pre-tokenization over tokenization algorithms and proposes GPE, a novel method that enhances tokenization for complex script languages.

Findings

Grapheme-based tokenization outperforms byte-level methods for complex scripts.

Pre-tokenization choice significantly impacts language fairness in models.

GPE improves representation quality for Tamil, Sinhala, and Hindi.

Abstract

Tokenizers act as a bridge between human language and the latent space of language models, influencing how language is represented in these models. Due to the immense popularity of English-Centric Large Language Models (LLMs), efforts are being made to adapt them for other languages. However, we demonstrate that, from a tokenization standpoint, not all tokenizers offer fair representation for complex script languages such as Tamil, Sinhala, and Hindi, primarily due to the choice of pre-tokenization methods. We go further to show that pre-tokenization plays a more critical role than the tokenization algorithm itself in achieving an egalitarian representation of these complex script languages. To address this, we introduce an improvement to the Byte Pair Encoding (BPE) algorithm by incorporating graphemes, which we term Grapheme Pair Encoding (GPE). Our experiments show that grapheme-based character extraction outperforms byte-level tokenizers for complex scripts. We validate this approach through experiments on Tamil, Sinhala, and Hindi.