Dictionaries to the Rescue: Cross-Lingual Vocabulary Transfer for Low-Resource Languages Using Bilingual Dictionaries

TL;DR

This paper introduces a simple, dictionary-based method for cross-lingual vocabulary transfer in low-resource languages, leveraging BPE tokenizer properties to improve language model adaptation.

Contribution

It proposes a novel vocabulary transfer technique using bilingual dictionaries and BPE tokenizer properties, outperforming existing methods for low-resource languages.

Findings

Outperforms existing methods in low-resource language adaptation

Effectively leverages bilingual dictionaries for vocabulary transfer

Demonstrates the utility of BPE tokenizer properties in transfer learning

Abstract

Cross-lingual vocabulary transfer plays a promising role in adapting pre-trained language models to new languages, including low-resource languages. Existing approaches that utilize monolingual or parallel corpora face challenges when applied to languages with limited resources. In this work, we propose a simple yet effective vocabulary transfer method that utilizes bilingual dictionaries, which are available for many languages, thanks to descriptive linguists. Our proposed method leverages a property of BPE tokenizers where removing a subword from the vocabulary causes a fallback to shorter subwords. The embeddings of target subwords are estimated iteratively by progressively removing them from the tokenizer. The experimental results show that our approach outperforms existing methods for low-resource languages, demonstrating the effectiveness of a dictionary-based approach for cross-lingual vocabulary transfer.