DocHop-QA: Towards Multi-Hop Reasoning over Multimodal Document Collections

TL;DR

DocHop-QA is a large-scale, multimodal, multi-document question answering benchmark based on scientific literature, designed to challenge and evaluate multi-hop reasoning capabilities of models across diverse formats and complex reasoning paths.

Contribution

It introduces a novel, domain-agnostic dataset with 11,379 instances, supporting open-ended, multimodal, multi-hop reasoning without relying on hyperlinks, and provides a comprehensive evaluation framework.

Findings

Demonstrates models' ability to perform complex multimodal reasoning

Shows the dataset's effectiveness in evaluating multi-hop QA

Highlights the need for advanced reasoning in scientific domains

Abstract

Despite recent advances in large language models (LLMs), most QA benchmarks are still confined to single-paragraph or single-document settings, failing to capture the complexity of real-world information-seeking tasks. Practical QA often requires multi-hop reasoning over information distributed across multiple documents, modalities, and structural formats. Although prior datasets made progress in this area, they rely heavily on Wikipedia-based content and unimodal plain text, with shallow reasoning paths that typically produce brief phrase-level or single-sentence answers, thus limiting their realism and generalizability. We propose DocHop-QA, a large-scale benchmark comprising 11,379 QA instances for multimodal, multi-document, multi-hop question answering. Constructed from publicly available scientific documents sourced from PubMed, DocHop-QA is domain-agnostic and incorporates diverse information formats, including textual passages, tables, and structural layout cues. Unlike existing datasets, DocHop-QA does not rely on explicitly hyperlinked documents; instead, it supports open-ended reasoning through semantic similarity and layout-aware evidence synthesis. To scale realistic QA construction, we designed an LLM-driven pipeline grounded in 11 high-frequency scientific question concepts. We evaluated DocHop-QA through four tasks spanning structured index prediction, generative answering, and multimodal integration, reflecting both discriminative and generative paradigms. These tasks demonstrate DocHop-QA's capacity to support complex, multimodal reasoning across multiple documents.