Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?

TL;DR

This paper introduces VocADT, a novel adapter-based method for vocabulary adaptation in pre-trained language models, which improves multilingual performance especially for Latin-script and fragmented languages without external resources.

Contribution

We propose VocADT, a scalable adapter-based approach for vocabulary adaptation that outperforms baselines across diverse languages and tasks without relying on external embeddings.

Findings

Latin-script languages benefit the most from adaptation.

Highly fragmented languages see significant improvements.

Vocabulary adaptation remains beneficial after fine-tuning for translation.

Abstract

Vocabulary adaptation, which integrates new vocabulary into pre-trained language models, enables expansion to new languages and mitigates token over-fragmentation. However, existing approaches are limited by their reliance on heuristics or external embeddings. We propose VocADT, a novel method for vocabulary adaptation using adapter modules that are trained to learn the optimal linear combination of existing embeddings while keeping the model's weights fixed. VocADT offers a flexible and scalable solution without depending on external resources or language constraints. Across 11 languages-with diverse scripts, resource availability, and fragmentation-we demonstrate that VocADT outperforms the original Mistral model and other baselines across various multilingual tasks including natural language understanding and machine translation. We find that Latin-script languages and highly fragmented languages benefit the most from vocabulary adaptation. We further fine-tune the adapted model on the generative task of machine translation and find that vocabulary adaptation is still beneficial after fine-tuning and that VocADT is the most effective.