Multilinguality Does not Make Sense: Investigating Factors Behind Zero-Shot Transfer in Sense-Aware Tasks

TL;DR

This paper challenges the assumption that multilingual training enhances zero-shot transfer in sense-aware NLP tasks, showing that other factors like data differences and artifacts are more influential.

Contribution

It provides a large-scale analysis across 28 languages demonstrating that multilinguality is not essential for effective transfer in sense-aware tasks.

Findings

Multilinguality is not necessary for zero-shot transfer in sense-aware tasks.

Differences in pretraining and fine-tuning data better explain transfer effectiveness.

The study offers empirical baselines and releases models for future research.

Abstract

Cross-lingual transfer is central to modern NLP, enabling models to perform tasks in languages different from those they were trained on. A common assumption is that training on more languages improves zero-shot transfer. We test this on sense-aware tasks-polysemy and lexical semantic change-and find that multilinguality is not necessary for effective transfer. Our large-scale analysis across 28 languages reveals that other factors, such as differences in pretraining and fine-tuning data and evaluation artifacts, better explain the perceived benefits of multilinguality. We also release fine-tuned models and provide empirical baselines to support future research. While focused on two sense-aware tasks, our findings offer broader insights into cross-lingual transfer, especially for low-resource languages.