Omanic: Towards Step-wise Evaluation of Multi-hop Reasoning in Large Language Models

TL;DR

Omanic introduces a multi-hop QA dataset with step-by-step reasoning annotations to evaluate and improve large language models' reasoning capabilities, revealing current limitations and enabling transfer learning.

Contribution

The paper presents Omanic, a new dataset with structural reasoning annotations, and demonstrates its effectiveness for evaluating and enhancing reasoning in large language models.

Findings

State-of-the-art LLMs achieve only 73.11% accuracy on OmanicBench.

Fine-tuning on OmanicSynth improves performance across reasoning benchmarks.

Stepwise analysis highlights the importance of factual completeness and error propagation.

Abstract

Reasoning-focused large language models (LLMs) have advanced in many NLP tasks, yet their evaluation remains challenging: final answers alone do not expose the intermediate reasoning steps, making it difficult to determine whether a model truly reasons correctly and where failures occur, while existing multi-hop QA benchmarks lack step-level annotations for diagnosing reasoning failures. To address this gap, we propose Omanic, an open-domain multi-hop QA resource that provides decomposed sub-questions and intermediate answers as structural annotations for analyzing reasoning processes. It contains 10,296 machine-generated training examples (OmanicSynth) and 967 expert-reviewed human-annotated evaluation examples (OmanicBench). Systematic evaluations show that state-of-the-art LLMs achieve only 73.11% multiple-choice accuracy on OmanicBench, confirming its high difficulty. Stepwise analysis reveals that CoT's performance hinges on factual completeness, with its gains diminishing under knowledge gaps and errors amplifying in later hops. Additionally, supervised fine-tuning on OmanicSynth brings substantial transfer gains (7.41 average points) across six reasoning and math benchmarks, validating the dataset's quality and further supporting the effectiveness of OmanicSynth as supervision for reasoning-capability transfer. We release the data at https://huggingface.co/datasets/li-lab/Omanic and the code at https://github.com/XiaojieGu/Omanic.