Interference Matrix: Quantifying Cross-Lingual Interference in Transformer Encoders

TL;DR

This study quantifies cross-lingual interference in multilingual Transformer models using an interference matrix, revealing asymmetrical patterns linked to script and aiding in optimizing model performance.

Contribution

We introduce a large-scale interference matrix for 83 languages, providing novel insights into cross-lingual interference patterns and their relation to script and model performance.

Findings

Interference is asymmetrical between language pairs.

Patterns do not align with linguistic family or embedding similarity.

Interference matrix predicts downstream task performance.

Abstract

In this paper, we present a comprehensive study of language interference in encoder-only Transformer models across 83 languages. We construct an interference matrix by training and evaluating small BERT-like models on all possible language pairs, providing a large-scale quantification of cross-lingual interference. Our analysis reveals that interference between languages is asymmetrical and that its patterns do not align with traditional linguistic characteristics, such as language family, nor with proxies like embedding similarity, but instead better relate to script. Finally, we demonstrate that the interference matrix effectively predicts performance on downstream tasks, serving as a tool to better design multilingual models to obtain optimal performance.