Is Sanskrit the most token-efficient language? A quantitative study using GPT, Gemini, and SentencePiece

TL;DR

This study quantifies Sanskrit's high token efficiency in LLMs compared to English and Hindi, revealing potential for cost savings and faster inference, and highlights the impact of tokenizer bias on non-English languages.

Contribution

It provides the first quantitative analysis of Sanskrit's token efficiency in modern tokenizers, demonstrating its compactness and implications for LLM cost and bias.

Findings

Sanskrit has approximately half the token count of English and Hindi.

Latest tokenizers reduce bias but do not fully capture Sanskrit's compactness.

Sanskrit's morphology offers significant potential for efficient language modeling.

Abstract

Tokens are the basic units of Large Language Models (LLMs). LLMs rely on tokenizers to segment text into these tokens, and tokenization is the primary determinant of computational and inference cost. Sanskrit, one of the oldest languages, is hypothesized to express more meaning per token due to its morphology and grammar rules; however, no prior work has quantified this. We use a dataset of 701 parallel verses of the Bhagavad Gita, which comprises three languages-Sanskrit, English, and Hindi along with transliteration of Sanskrit into English. We test tokenizers including SentencePiece (SPM), older GPT models, and the latest generation tokenizers from Gemini and GPT. We use metrics of token count, characters per token (token efficiency), and tokens per character (token cost). Results show a ~2x difference in token counts between Sanskrit and English/Hindi under the unbiased SPM baseline. English/Hindi translations of Sanskrit commentary resulted in an approximately 20x increase in token count. GPT o200k base (latest, used by GPT-4o) and Gemini (latest) reduce bias by a significant degree compared to GPT cl100k base (used until GPT-4), but still fail to fully capture Sanskrit's compactness. This matters because there might be a penalty bias for non-English users, which inflates the token count. This research provides a foundation for improving future tokenizer design and shows the potential of Sanskrit for highly compact encoding, saving on cost while speeding up training and inference. The code and dataset are available at https://github.com/anshulkr713/sanskrit-token-efficiency