On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment

TL;DR

This paper systematically studies negative interference in multilingual models, revealing its impact on both high-resource and low-resource languages, and proposes a meta-learning approach with language-specific layers to mitigate this issue.

Contribution

It is the first to systematically analyze negative interference in multilingual models and introduces a meta-learning method with language-specific parameters to improve transferability.

Findings

Negative interference affects both high-resource and low-resource languages.

Language-specific parameters can help mitigate negative interference.

Meta-learning with language-specific layers improves cross-lingual transfer.

Abstract

Modern multilingual models are trained on concatenated text from multiple languages in hopes of conferring benefits to each (positive transfer), with the most pronounced benefits accruing to low-resource languages. However, recent work has shown that this approach can degrade performance on high-resource languages, a phenomenon known as negative interference. In this paper, we present the first systematic study of negative interference. We show that, contrary to previous belief, negative interference also impacts low-resource languages. While parameters are maximally shared to learn language-universal structures, we demonstrate that language-specific parameters do exist in multilingual models and they are a potential cause of negative interference. Motivated by these observations, we also present a meta-learning algorithm that obtains better cross-lingual transferability and alleviates negative interference, by adding language-specific layers as meta-parameters and training them in a manner that explicitly improves shared layers' generalization on all languages. Overall, our results show that negative interference is more common than previously known, suggesting new directions for improving multilingual representations.