Do Language Models Reason Across Languages?

TL;DR

This paper investigates whether multilingual language models can perform step-by-step reasoning across languages, revealing limitations in their reasoning process and proposing a prompting method to improve multi-hop question answering accuracy.

Contribution

It introduces a two-hop multilingual reasoning setting, analyzes models' reasoning failures, and proposes a SUBQ prompting method to enhance multi-step reasoning accuracy.

Findings

Models are more sensitive to language variation in answer documents.

Up to 33% of cases show models fail to infer bridging info but still answer correctly.

SUBQ prompting improves multi-hop reasoning accuracy from 10.1% to 66.5%.

Abstract

The real-world information sources are inherently multilingual, which naturally raises a question about whether language models can synthesize information across languages. In this paper, we introduce a simple two-hop question answering setting, where answering a question requires making inferences over two multilingual documents. We find that language models are more sensitive to language variation in answer-span documents than in those providing bridging information, despite the equal importance of both documents for answering a question. Under a step-by-step sub-question evaluation, we further show that in up to 33% of multilingual cases, models fail to infer the bridging information in the first step yet still answer the overall question correctly. This indicates that reasoning in language models, especially in multilingual settings, does not follow a faithful step-by-step decomposition. Subsequently, we show that the absence of reasoning decomposition leads to around 18% composition failure, where both sub-questions are answered correctly but fail for the final two-hop questions. To mitigate this, we propose a simple three-stage SUBQ prompting method to guide the multi-step reasoning with sub-questions, which boosts accuracy from 10.1% to 66.5%.