Improving Multilingual Models with Language-Clustered Vocabularies

TL;DR

This paper introduces a new multilingual vocabulary generation method that clusters languages and combines their vocabularies, improving performance and reducing out-of-vocabulary issues without increasing model size.

Contribution

The paper proposes a novel language-clustered vocabulary generation procedure for multilingual models, balancing cross-lingual sharing and language-specific vocabularies.

Findings

Improved performance on TyDi QA (+2.9 F1)

Enhanced XNLI accuracy (+2.1%)

Reduced out-of-vocabulary rate by a factor of 8

Abstract

State-of-the-art multilingual models depend on vocabularies that cover all of the languages the model will expect to see at inference time, but the standard methods for generating those vocabularies are not ideal for massively multilingual applications. In this work, we introduce a novel procedure for multilingual vocabulary generation that combines the separately trained vocabularies of several automatically derived language clusters, thus balancing the trade-off between cross-lingual subword sharing and language-specific vocabularies. Our experiments show improvements across languages on key multilingual benchmark tasks TyDi QA (+2.9 F1), XNLI (+2.1\%), and WikiAnn NER (+2.8 F1) and factor of 8 reduction in out-of-vocabulary rate, all without increasing the size of the model or data.