Probing Multilingual Language Models for Discourse

TL;DR

This paper evaluates how well pre-trained multilingual language models transfer discourse-level knowledge across languages, revealing strengths of XLM-RoBERTa and insights into factors affecting cross-lingual transfer.

Contribution

It provides a systematic evaluation of multilingual models on discourse tasks across many languages, highlighting the performance of XLM-RoBERTa and effects of model distillation.

Findings

XLM-RoBERTa models perform best on discourse tasks across languages.

Model distillation may impair cross-lingual transfer of sentence representations.

Language dissimilarity has only a modest impact on transfer performance.

Abstract

Pre-trained multilingual language models have become an important building block in multilingual natural language processing. In the present paper, we investigate a range of such models to find out how well they transfer discourse-level knowledge across languages. This is done with a systematic evaluation on a broader set of discourse-level tasks than has been previously been assembled. We find that the XLM-RoBERTa family of models consistently show the best performance, by simultaneously being good monolingual models and degrading relatively little in a zero-shot setting. Our results also indicate that model distillation may hurt the ability of cross-lingual transfer of sentence representations, while language dissimilarity at most has a modest effect. We hope that our test suite, covering 5 tasks with a total of 22 languages in 10 distinct families, will serve as a useful evaluation platform for multilingual performance at and beyond the sentence level.