Factual Consistency of Multilingual Pretrained Language Models

TL;DR

This paper investigates the factual consistency of multilingual pretrained language models like mBERT and XLM-R, revealing high inconsistency across languages and similar issues as monolingual models, impacting their reliability for knowledge-based tasks.

Contribution

The study introduces mParaRel, a resource for analyzing multilingual factual consistency, and provides a comparative analysis of multilingual models' consistency across 45 languages.

Findings

mBERT is as inconsistent as English BERT in English paraphrases

Both mBERT and XLM-R show high inconsistency in English

Inconsistency increases across 45 languages

Abstract

Pretrained language models can be queried for factual knowledge, with potential applications in knowledge base acquisition and tasks that require inference. However, for that, we need to know how reliable this knowledge is, and recent work has shown that monolingual English language models lack consistency when predicting factual knowledge, that is, they fill-in-the-blank differently for paraphrases describing the same fact. In this paper, we extend the analysis of consistency to a multilingual setting. We introduce a resource, mParaRel, and investigate (i) whether multilingual language models such as mBERT and XLM-R are more consistent than their monolingual counterparts; and (ii) if such models are equally consistent across languages. We find that mBERT is as inconsistent as English BERT in English paraphrases, but that both mBERT and XLM-R exhibit a high degree of inconsistency in English and even more so for all the other 45 languages.