Do Multi-Lingual Pre-trained Language Models Reveal Consistent Token Attributions in Different Languages?

TL;DR

This paper investigates whether multi-lingual pre-trained language models assign consistent token attributions across languages, revealing significant differences and correlations with downstream task performance.

Contribution

It introduces the CCTA framework to evaluate cross-lingual token attribution consistency and provides empirical insights into factors affecting this consistency.

Findings

Multi-lingual PLMs show significant attribution differences for synonyms across languages.

Spanish training data leads to the most consistent token attributions.

Token attribution consistency correlates strongly with downstream task performance.

Abstract

During the past several years, a surge of multi-lingual Pre-trained Language Models (PLMs) has been proposed to achieve state-of-the-art performance in many cross-lingual downstream tasks. However, the understanding of why multi-lingual PLMs perform well is still an open domain. For example, it is unclear whether multi-Lingual PLMs reveal consistent token attributions in different languages. To address this, in this paper, we propose a Cross-lingual Consistency of Token Attributions (CCTA) evaluation framework. Extensive experiments in three downstream tasks demonstrate that multi-lingual PLMs assign significantly different attributions to multi-lingual synonyms. Moreover, we have the following observations: 1) the Spanish achieves the most consistent token attributions in different languages when it is used for training PLMs; 2) the consistency of token attributions strongly correlates with performance in downstream tasks.