Evaluation of contextual embeddings on less-resourced languages

TL;DR

This paper compares the performance of various contextual embedding models like ELMo and BERT across nine languages and 14 tasks, highlighting the strengths of monolingual BERT models and the effectiveness of multilingual models in cross-lingual settings.

Contribution

First comprehensive multilingual empirical comparison of ELMo and BERT models on diverse NLP tasks in less-resourced languages.

Findings

Monolingual BERT models generally outperform ELMo in monolingual settings.

In dependency parsing, ELMo models trained on large corpora are more competitive.

Multilingual BERT models trained on few languages perform well in cross-lingual tasks.

Abstract

The current dominance of deep neural networks in natural language processing is based on contextual embeddings such as ELMo, BERT, and BERT derivatives. Most existing work focuses on English; in contrast, we present here the first multilingual empirical comparison of two ELMo and several monolingual and multilingual BERT models using 14 tasks in nine languages. In monolingual settings, our analysis shows that monolingual BERT models generally dominate, with a few exceptions such as the dependency parsing task, where they are not competitive with ELMo models trained on large corpora. In cross-lingual settings, BERT models trained on only a few languages mostly do best, closely followed by massively multilingual BERT models.