OMHBench: Benchmarking Balanced and Grounded Omni-Modal Multi-Hop Reasoning

TL;DR

OMHBench is a new benchmark for evaluating multi-hop reasoning across text, vision, and speech in multimodal large language models, addressing limitations of previous evaluation methods.

Contribution

It introduces a balanced, grounded omni-modal reasoning benchmark with extensive evaluation of state-of-the-art models highlighting current challenges.

Findings

Significant performance gap between proprietary and open-source models.

Models are highly sensitive to reasoning path variations.

Models struggle with processing speech modality.

Abstract

Multimodal Large Language Models (MLLMs) have increasingly supported omni-modal processing across text, vision, and speech. However, existing evaluation frameworks for such models suffer from critical limitations, including modality shortcuts and biased reasoning paths. To address these challenges, we propose OMHBench, a novel benchmark designed to rigorously evaluate omni-modal multi-hop reasoning. It consists of 6,144 questions with balanced reasoning paths that are jointly grounded across all three modalities. Extensive evaluation of 13 state-of-the-art models reveals that (1) a large performance gap exists between proprietary and open-source MLLMs and (2) even proprietary models exhibit high sensitivity to reasoning path variations, resulting in asymmetric omni-modal grounding. Notably, models struggle when processing the speech modality, underscoring the need for balanced, multi-hop evaluation of omni-modal intelligence.