Unlocking Multilingual Reasoning Capability of LLMs and LVLMs through Representation Engineering

TL;DR

This paper introduces a training-free inference method called MRRE that improves multilingual reasoning in LLMs and LVLMs by injecting precomputed vectors during inference, enhancing performance in low-resource languages without additional training or translation tools.

Contribution

The paper proposes a novel, training-free inference technique that enhances multilingual reasoning by injecting precomputed vectors, avoiding costly training or external translation tools.

Findings

Average reasoning performance improved by 5.48%.

Up to 7.54% improvement in low-resource languages.

Input-output language consistency increased by 3.78%.

Abstract

Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) demonstrate strong reasoning capabilities, yet their performance in English significantly outperforms that in low-resource languages, raising fairness concerns in multilingual applications. Existing approaches either rely on costly multilingual training or employ prompting with external translation tools, both of which are resource-intensive and sensitive to translation quality. To address these limitations, we propose a training-free inference-time method to enhance Multilingual Reasoning capabilities via Representation Engineering (MRRE) without using any additional training data or tools. MRRE sequentially injects two precomputed vectors at specific layers during inference processing: cross-lingual reasoning enhancement vectors, which steer non-English reasoning representations toward English space to unlock multilingual reasoning, and target-language output anchoring vectors, which restore the distribution of the target language to preserve input-output language consistency. Comprehensive experiments across six advanced LLMs and LVLMs on four reasoning benchmarks demonstrate that MRRE consistently enhances non-English reasoning by an average gain of 5.48% and up to 7.54% in low-resource languages (Thai and Swahili), while improving input-output language consistency by 3.78%.