Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledge

TL;DR

This paper investigates how multilingual language models acquire and represent factual knowledge across languages, identifying patterns of knowledge transfer and the challenges in maintaining consistency in facts.

Contribution

It introduces methods to differentiate independent, shared, and transferred factual knowledge in ML-LMs, providing insights into their knowledge acquisition processes.

Findings

Identified three patterns of fact acquisition: language-independent, shared, and transferred.

Demonstrated the difficulty of maintaining consistent facts across languages.

Provided methods to trace the roots of factual knowledge in ML-LMs.

Abstract

Acquiring factual knowledge for language models (LMs) in low-resource languages poses a serious challenge, thus resorting to cross-lingual transfer in multilingual LMs (ML-LMs). In this study, we ask how ML-LMs acquire and represent factual knowledge. Using the multilingual factual knowledge probing dataset, mLAMA, we first conducted a neuron investigation of ML-LMs (specifically, multilingual BERT). We then traced the roots of facts back to the knowledge source (Wikipedia) to identify the ways in which ML-LMs acquire specific facts. We finally identified three patterns of acquiring and representing facts in ML-LMs: language-independent, cross-lingual shared and transferred, and devised methods for differentiating them. Our findings highlight the challenge of maintaining consistent factual knowledge across languages, underscoring the need for better fact representation learning in ML-LMs.