MMErroR: A Benchmark for Erroneous Reasoning in Vision-Language Models

TL;DR

MMErroR introduces a comprehensive benchmark to evaluate vision-language models' ability to detect and classify reasoning errors across diverse multi-modal tasks.

Contribution

The paper presents MMErroR, a large-scale, error-centric benchmark for assessing models' reasoning error detection in vision-language tasks, highlighting current limitations.

Findings

Even the best model correctly classifies errors only 66.65% of the time.

MMErroR covers 24 subdomains across six top-level domains.

Models struggle to reliably detect and classify reasoning errors.

Abstract

Recent advances in Vision-Language Models (VLMs) have improved performance in multi-modal learning, raising the question of whether these models truly understand the content they process. Crucially, can VLMs detect when a reasoning process is wrong and identify its error type? To answer this, we present MMErroR, a multi-modal benchmark of 1997 samples, each embedding a single coherent reasoning error. These samples span 24 subdomains across six top-level domains, ensuring broad coverage and taxonomic richness. Unlike existing benchmarks that focus on answer correctness, MMErroR targets a process-level, error-centric evaluation that requires models to detect incorrect reasoning and classify the error type within both visual and linguistic contexts. We evaluate 12 representative VLMs, and even the best model, Gemini-3-Pro-Preview, classifies the error correctly in only 66.65\% of cases, underscoring the challenge of identifying erroneous reasoning. Furthermore, the ability to accurately identify errors offers valuable insights into the capabilities of multi-modal models. Project Page: https://mmerror-benchmark.github.io