Linguistic Neuron Overlap Patterns to Facilitate Cross-lingual Transfer on Low-resource Languages

TL;DR

This paper introduces BridgeX-ICL, a neuron-sharing method that enhances zero-shot cross-lingual learning in large language models, especially benefiting low-resource languages by leveraging linguistic neuron overlaps.

Contribution

It proposes a novel neuron-sharing approach, BridgeX-ICL, guided by an HSIC-based metric, to improve cross-lingual transfer in LLMs without additional fine-tuning.

Findings

BridgeX-ICL improves performance on 4 cross-lingual tasks.

Neuron overlap patterns correlate with transfer success.

Method is effective across diverse language pairs.

Abstract

The current Large Language Models (LLMs) face significant challenges in improving their performance on low-resource languages and urgently need data-efficient methods without costly fine-tuning. From the perspective of language-bridge, we propose a simple yet effective method, namely BridgeX-ICL, to improve the zero-shot Cross-lingual In-Context Learning (X-ICL) for low-resource languages. Unlike existing works focusing on language-specific neurons, BridgeX-ICL explores whether sharing neurons can improve cross-lingual performance in LLMs. We construct neuron probe data from the ground-truth MUSE bilingual dictionaries, and define a subset of language overlap neurons accordingly to ensure full activation of these anchored neurons. Subsequently, we propose an HSIC-based metric to quantify LLMs' internal linguistic spectrum based on overlapping neurons, guiding optimal bridge selection. The experiments conducted on 4 cross-lingual tasks and 15 language pairs from 7 diverse families, covering both high-low and moderate-low pairs, validate the effectiveness of BridgeX-ICL and offer empirical insights into the underlying multilingual mechanisms of LLMs. The code is publicly available at https://github.com/xuyuemei/BridgeX-ICL.