Failure Modes in Multi-Hop QA: The Weakest Link Effect and the Recognition Bottleneck

TL;DR

This paper investigates why large language models struggle with multi-hop reasoning, identifying the weakest link effect and proposing a semantic probing method to improve evidence recognition and integration.

Contribution

It introduces Multi-Focus Attention Instruction (MFAI) to disentangle recognition and synthesis failures, revealing the weakest link effect and demonstrating improved reasoning accuracy.

Findings

Multi-hop performance is dominated by the least visible evidence.

MFAI improves accuracy by up to 11.49% in low-visibility positions.

System-2 reasoning models effectively locate and integrate evidence even in noisy, long contexts.

Abstract

Despite scaling to massive context windows, Large Language Models (LLMs) struggle with multi-hop reasoning due to inherent position bias, which causes them to overlook information at certain positions. Whether these failures stem from an inability to locate evidence (recognition failure) or integrate it (synthesis failure) is unclear. We introduce Multi-Focus Attention Instruction (MFAI), a semantic probe to disentangle these mechanisms by explicitly steering attention towards selected positions. Across 5 LLMs on two multi-hop QA tasks (MuSiQue and NeoQA), we identify the "Weakest Link Effect": in our 18-document, 3-bucket setting, multi-hop reasoning performance collapses to the level of the least visible evidence, governed by absolute position rather than the linear distance between facts. While matched MFAI resolves recognition bottlenecks, improving accuracy by up to 11.49% in low-visibility positions, misleading MFAI yields divergent effects modulated by task topology: entity-centric tasks with vertical reasoning chains are vulnerable, whereas event-centric tasks with horizontal evidence structures are more resilient. Finally, we demonstrate that thinking models utilizing System-2 reasoning effectively locate and integrate the required information, matching gold-only baselines even in noisy, long-context settings. Supplementary experiments on 2WikiMultiHopQA, extended 3-4 hop counts, and a 32B model confirm these findings generalize across datasets, reasoning depths, and model scales.