Bridging the Language Gaps in Large Language Models with Inference-Time Cross-Lingual Intervention

TL;DR

This paper introduces INCLINE, a cost-effective inference-time method that improves low-resource language performance in large language models by aligning their internal representations with high-resource languages without retraining.

Contribution

The paper presents INCLINE, a novel inference-time framework for cross-lingual alignment that enhances LLM performance on low-resource languages without additional training.

Findings

INCLINE significantly improves multilingual task performance.

The method is highly cost-effective and widely applicable.

Experimental results outperform recent baselines.

Abstract

Large Language Models (LLMs) have shown remarkable capabilities in natural language processing but exhibit significant performance gaps among different languages. Most existing approaches to address these disparities rely on pretraining or fine-tuning, which are resource-intensive. To overcome these limitations without incurring significant costs, we propose Inference-Time Cross-Lingual Intervention (INCLINE), a novel framework that enhances LLM performance on low-performing (source) languages by aligning their internal representations with those of high-performing (target) languages during inference. INCLINE initially learns alignment matrices using parallel sentences from source and target languages through a Least-Squares optimization, and then applies these matrices during inference to transform the low-performing language representations toward the high-performing language space. Extensive experiments on nine benchmarks with five LLMs demonstrate that INCLINE significantly improves performance across diverse tasks and languages, compared to recent strong baselines. Our analysis demonstrates that INCLINE is highly cost-effective and applicable to a wide range of applications. In addition, we release the code to foster research along this line: https://github.com/weixuan-wang123/INCLINE.