MultiHoax: A Dataset of Multi-hop False-Premise Questions

TL;DR

MultiHoax is a new benchmark dataset designed to evaluate large language models' ability to detect false premises in complex, multi-hop reasoning scenarios across diverse knowledge domains and countries.

Contribution

We introduce MultiHoax, the first benchmark focusing on multi-hop false-premise questions, highlighting the challenges LLMs face in multi-step factual reasoning and false premise detection.

Findings

State-of-the-art LLMs struggle with multi-hop false-premise detection.

Models show limited performance across different countries and knowledge categories.

Multi-hop reasoning remains a significant challenge for current LLMs.

Abstract

As Large Language Models are increasingly deployed in high-stakes domains, their ability to detect false assumptions and reason critically is crucial for ensuring reliable outputs. False-premise questions (FPQs) serve as an important evaluation method by exposing cases where flawed assumptions lead to incorrect responses. While existing benchmarks focus on single-hop FPQs, real-world reasoning often requires multi-hop inference, where models must verify consistency across multiple reasoning steps rather than relying on surface-level cues. To address this gap, we introduce MultiHoax, a benchmark for evaluating LLMs' ability to handle false premises in complex, multi-step reasoning tasks. Our dataset spans seven countries and ten diverse knowledge categories, using Wikipedia as the primary knowledge source to enable factual reasoning across regions. Experiments reveal that state-of-the-art LLMs struggle to detect false premises across different countries, knowledge categories, and multi-hop reasoning types, highlighting the need for improved false premise detection and more robust multi-hop reasoning capabilities in LLMs.