NovelHopQA: Diagnosing Multi-Hop Reasoning Failures in Long Narrative Contexts

TL;DR

NovelHopQA introduces a comprehensive benchmark for evaluating multi-hop question answering over extremely long narrative texts, revealing that larger models still struggle with reasoning across extended contexts.

Contribution

It is the first benchmark to jointly vary context length and reasoning depth in long narratives, providing a new diagnostic tool for multi-hop reasoning in large language models.

Findings

Accuracy drops with increased hops and context length

Scale alone does not ensure robust reasoning

Common failure modes include missed final-hop integration

Abstract

Current large language models (LLMs) struggle to answer questions that span tens of thousands of tokens, especially when multi-hop reasoning is involved. While prior benchmarks explore long-context comprehension or multi-hop reasoning in isolation, none jointly vary context length and reasoning depth in natural narrative settings. We introduce NovelHopQA, the first benchmark to evaluate 1-4 hop QA over 64k-128k-token excerpts from 83 full-length public-domain novels. A keyword-guided pipeline builds hop-separated chains grounded in coherent storylines. We evaluate seven state-of-the-art models and apply oracle-context filtering to ensure all questions are genuinely answerable. Human annotators validate both alignment and hop depth. We additionally present retrieval-augmented generation (RAG) evaluations to test model performance when only selected passages are provided instead of the full context. We noticed consistent accuracy drops with increased hops and context length increase, even for frontier models-revealing that sheer scale does not guarantee robust reasoning. Failure-mode analysis highlights common breakdowns such as missed final-hop integration and long-range drift. NovelHopQA offers a controlled diagnostic setting to test multi-hop reasoning at scale. All code and datasets are available at https://novelhopqa.github.io.