Towards Fully Bilingual Deep Language Modeling

TL;DR

This paper demonstrates that it is feasible to pre-train a fully bilingual deep language model for two remotely related languages without sacrificing performance, achieving results comparable to monolingual models by increasing vocabulary size.

Contribution

The study shows that a bilingual BERT model can match monolingual performance on both languages, challenging the notion that multilinguality necessarily reduces monolingual effectiveness.

Findings

Bilingual BERT performs on par with English BERT on GLUE.

It nearly matches Finnish BERT on Finnish NLP tasks.

Increasing vocabulary size enables learning two languages effectively.

Abstract

Language models based on deep neural networks have facilitated great advances in natural language processing and understanding tasks in recent years. While models covering a large number of languages have been introduced, their multilinguality has come at a cost in terms of monolingual performance, and the best-performing models at most tasks not involving cross-lingual transfer remain monolingual. In this paper, we consider the question of whether it is possible to pre-train a bilingual model for two remotely related languages without compromising performance at either language. We collect pre-training data, create a Finnish-English bilingual BERT model and evaluate its performance on datasets used to evaluate the corresponding monolingual models. Our bilingual model performs on par with Google's original English BERT on GLUE and nearly matches the performance of monolingual Finnish BERT on a range of Finnish NLP tasks, clearly outperforming multilingual BERT. We find that when the model vocabulary size is increased, the BERT-Base architecture has sufficient capacity to learn two remotely related languages to a level where it achieves comparable performance with monolingual models, demonstrating the feasibility of training fully bilingual deep language models. The model and all tools involved in its creation are freely available at https://github.com/TurkuNLP/biBERT