Problematic Tokens: Tokenizer Bias in Large Language Models

TL;DR

This paper investigates how tokenization biases in large language models like GPT-4o contribute to disparities in language processing, highlighting security and ethical concerns stemming from inadequate token vocabularies for under-resourced languages.

Contribution

It analyzes the tokenization process of GPT-4o, revealing how token vocabulary construction leads to biases and proposing strategies to improve tokenization for fairness and security.

Findings

Tokenization biases affect non-English language performance.

Simplified token handling amplifies security and ethical risks.

Proposed solutions mitigate biases and improve tokenization robustness.

Abstract

Recent advancements in large language models(LLMs), such as GPT-4 and GPT-4o, have shown exceptional performance, especially in languages with abundant resources like English, thanks to extensive datasets that ensure robust training. Conversely, these models exhibit limitations when processing under-resourced languages such as Chinese and Korean, where issues including hallucinatory responses remain prevalent. This paper traces the roots of these disparities to the tokenization process inherent to these models. Specifically, it explores how the tokenizers vocabulary, often used to speed up the tokenization process and reduce tokens but constructed independently of the actual model training data, inadequately represents non-English languages. This misrepresentation results in the propagation of under-trained or untrained tokens, which perpetuate biases and pose serious concerns related to data security and ethical standards. We aim to dissect the tokenization mechanics of GPT-4o, illustrating how its simplified token-handling methods amplify these risks and offer strategic solutions to mitigate associated security and ethical issues. Through this study, we emphasize the critical need to rethink tokenization frameworks to foster more equitable and secure AI technologies. The code and data are available at https://github.com/yeyimilk/LLMGPT4o