Are Knowledge and Reference in Multilingual Language Models Cross-Lingually Consistent?

TL;DR

This paper investigates the cross-lingual consistency of factual knowledge in multilingual language models, analyzing factors affecting it and proposing strategies like code-switching and alignment to improve consistency.

Contribution

It provides a comprehensive analysis of cross-lingual consistency, evaluates various models and techniques, and highlights effective strategies for enhancing multilingual factual knowledge transfer.

Findings

Cross-lingual consistency varies across languages and model layers.

Code-switching training improves cross-lingual consistency.

Cross-lingual alignment supervision enhances model performance.

Abstract

Cross-lingual consistency should be considered to assess cross-lingual transferability, maintain the factuality of the model knowledge across languages, and preserve the parity of language model performance. We are thus interested in analyzing, evaluating, and interpreting cross-lingual consistency for factual knowledge. To facilitate our study, we examine multiple pretrained models and tuned models with code-mixed coreferential statements that convey identical knowledge across languages. Interpretability approaches are leveraged to analyze the behavior of a model in cross-lingual contexts, showing different levels of consistency in multilingual models, subject to language families, linguistic factors, scripts, and a bottleneck in cross-lingual consistency on a particular layer. Code-switching training and cross-lingual word alignment objectives show the most promising results, emphasizing the worthiness of cross-lingual alignment supervision and code-switching strategies for both multilingual performance and cross-lingual consistency enhancement. In addition, experimental results suggest promising result for calibrating consistency in the test time via activation patching.