Tokenizations for Austronesian Language Models: study on languages in Indonesia Archipelago

TL;DR

This paper introduces a syllable-based tokenization method inspired by traditional Indonesian scripts, improving linguistic alignment and consistency across Austronesian languages for large language models.

Contribution

It develops a novel syllable-based tokenization framework tailored for Austronesian languages, addressing limitations of subword methods optimized on English corpora.

Findings

Syllable-based tokenization yields consistent TPC across languages.

It increases token sequence similarity by approximately 21% over GPT-2.

The approach better preserves phonological and morphological patterns.

Abstract

Tokenization constitutes a fundamental stage in Large Language Model (LLM) processing; however, subword-based tokenization methods optimized on English-dominant corpora may produce token fragmentation misaligned with the linguistic structures of Austronesian languages. This study aimed to develop a syllable-based tokenization framework adopting principles from traditional Indonesian scripts (aksara) for regional languages of Indonesia. A syllabic segmentation procedure was constructed based on the logic of abugida writing systems and implemented with a vocabulary of 2,843 tokens extracted from the Indonesian dictionary (KBBI). Evaluation was conducted on the NusaX dataset comprising 1,000 parallel translation samples across 10 regional languages, Indonesian, and English. Analysis employed Token per Character (TPC) ratio and sequence alignment using the Smith-Waterman algorithm. Results demonstrated that syllable-based tokenization yielded consistent TPC values across all regional languages, whereas GPT-2 exhibited an inverse pattern with the lowest TPC for English. Syllable-based tokenization consistently produced higher token sequence similarity scores, with an average increase of approximately 21% compared to GPT-2. These findings confirm that the syllable-based approach more effectively preserves phonological and morphological patterns across related Austronesian languages, offering a linguistically principled foundation for multilingual LLM development.