Unveiling the Influence of Amplifying Language-Specific Neurons

TL;DR

This paper explores how amplifying language-specific neurons in multilingual models affects language steering and downstream task performance, revealing benefits for low-resource languages but limited cross-lingual transfer improvements.

Contribution

It introduces a method to amplify language-specific neurons and evaluates its impact across multiple languages and tasks, highlighting their role in multilingual model behavior.

Findings

Amplification effectively steers models toward target languages.

Improves performance on some downstream tasks for low-resource languages.

Degrades cross-language transfer in most cases.

Abstract

Language-specific neurons in LLMs that strongly correlate with individual languages have been shown to influence model behavior by deactivating them. However, their role in amplification remains underexplored. This work investigates the effect of amplifying language-specific neurons through interventions across 18 languages, including low-resource ones, using three models primarily trained in different languages. We compare amplification factors by their effectiveness in steering to the target language using a proposed Language Steering Shift (LSS) evaluation score, then evaluate it on downstream tasks: commonsense reasoning (XCOPA, XWinograd), knowledge (Include), and translation (FLORES). The optimal amplification factors effectively steer output toward nearly all tested languages. Intervention using this factor on downstream tasks improves self-language performance in some cases but generally degrades cross-language results. These findings highlight the effect of language-specific neurons in multilingual behavior, where amplification can be beneficial especially for low-resource languages, but provides limited advantage for cross-lingual transfer.