CRANE: Causal Relevance Analysis of Language-Specific Neurons in Multilingual Large Language Models

TL;DR

CRANE introduces a relevance-based framework to identify language-specific neurons in multilingual LLMs, revealing that these neurons are language-selective but not exclusive, and outperform activation-based methods in isolating language components.

Contribution

The paper presents CRANE, a novel neuron-level analysis method that redefines language specificity through functional necessity, improving the identification of language-specific neurons in multilingual models.

Findings

Neuron interventions show language-relevant neurons selectively affect target language performance.

CRANE outperforms activation-based methods in isolating language-specific components.

Language-specific neurons are mostly language-selective but not exclusive.

Abstract

Multilingual large language models (LLMs) achieve strong performance across languages, yet how language capabilities are organized at the neuron level remains poorly understood. Prior work has identified language-related neurons mainly through activation-based heuristics, which conflate language preference with functional importance. We propose CRANE, a relevance-based analysis framework that redefines language specificity in terms of functional necessity, identifying language-specific neurons through targeted neuron-level interventions. CRANE characterizes neuron specialization by their contribution to language-conditioned predictions rather than activation magnitude. Our implementation will be made publicly available. Neuron-level interventions reveal a consistent asymmetric pattern: masking neurons relevant to a target language selectively degrades performance on that language while preserving performance on other languages to a substantial extent, indicating language-selective but non-exclusive neuron specializations. Experiments on English, Chinese, and Vietnamese across multiple benchmarks, together with a dedicated relevance-based metric and base-to-chat model transfer analysis, show that CRANE isolates language-specific components more precisely than activation-based methods.