Is Multilingual BERT Fluent in Language Generation?

TL;DR

This paper evaluates multilingual BERT's capabilities across various languages and tasks, revealing it underperforms compared to monolingual models, especially for Nordic languages, and is less effective for language generation.

Contribution

The study provides a comprehensive assessment of multilingual BERT's performance on syntactic, cloze, and generation tasks across multiple languages, highlighting its limitations.

Findings

Multilingual BERT underperforms compared to monolingual models.

English and German models perform well at generation tasks.

Multilingual BERT struggles with Nordic languages.

Abstract

The multilingual BERT model is trained on 104 languages and meant to serve as a universal language model and tool for encoding sentences. We explore how well the model performs on several languages across several tasks: a diagnostic classification probing the embeddings for a particular syntactic property, a cloze task testing the language modelling ability to fill in gaps in a sentence, and a natural language generation task testing for the ability to produce coherent text fitting a given context. We find that the currently available multilingual BERT model is clearly inferior to the monolingual counterparts, and cannot in many cases serve as a substitute for a well-trained monolingual model. We find that the English and German models perform well at generation, whereas the multilingual model is lacking, in particular, for Nordic languages.