Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training

TL;DR

This paper introduces VoCap, an algorithm for allocating vocabulary capacity across languages in cross-lingual models, and proposes k-NN-based sampling to speed up training while maintaining performance.

Contribution

The paper presents VoCap for dynamic vocabulary allocation and a k-NN-based sampling method to accelerate training in cross-lingual models.

Findings

VoCap improves vocabulary representation for under-represented languages.

k-NN-based sampling speeds up pre-training without performance loss.

Multilingual vocabularies learned with VoCap enhance cross-lingual model performance.

Abstract

Compared to monolingual models, cross-lingual models usually require a more expressive vocabulary to represent all languages adequately. We find that many languages are under-represented in recent cross-lingual language models due to the limited vocabulary capacity. To this end, we propose an algorithm VoCap to determine the desired vocabulary capacity of each language. However, increasing the vocabulary size significantly slows down the pre-training speed. In order to address the issues, we propose k-NN-based target sampling to accelerate the expensive softmax. Our experiments show that the multilingual vocabulary learned with VoCap benefits cross-lingual language model pre-training. Moreover, k-NN-based target sampling mitigates the side-effects of increasing the vocabulary size while achieving comparable performance and faster pre-training speed. The code and the pretrained multilingual vocabularies are available at https://github.com/bozheng-hit/VoCapXLM.