Beyond Accuracy: Evaluating Grounded Visual Evidence in Thinking with Images

TL;DR

This paper introduces ViEBench, a new benchmark for evaluating visual reasoning in vision-language models, focusing on faithfulness and explainability beyond just accuracy, with detailed diagnostics and expert-annotated visual evidence.

Contribution

It presents ViEBench, a process-verifiable benchmark with multi-scenario images and a dual-axis evaluation matrix to assess visual reasoning and grounding fidelity in VLMs.

Findings

VLMs sometimes produce correct answers without relevant evidence.

Models can locate evidence but fail to use it effectively.

ViEBench enables transparent diagnosis of model reasoning behaviors.

Abstract

Despite the remarkable progress of Vision-Language Models (VLMs) in adopting "Thinking-with-Images" capabilities, accurately evaluating the authenticity of their reasoning process remains a critical challenge. Existing benchmarks mainly rely on outcome-oriented accuracy, lacking the capability to assess whether models can accurately leverage fine-grained visual cues for multi-step reasoning. To address these limitations, we propose ViEBench, a process-verifiable benchmark designed to evaluate faithful visual reasoning. Comprising 200 multi-scenario high-resolution images with expert-annotated visual evidence, ViEBench uniquely categorizes tasks by difficulty into perception and reasoning dimensions, where reasoning tasks require utilizing localized visual details with prior knowledge. To establish comprehensive evaluation criteria, we introduce a dual-axis matrix that provides fine-grained metrics through four diagnostic quadrants, enabling transparent diagnosis of model behavior across varying task complexities. Our experiments yield several interesting observations: (1) VLMs can sometimes produce correct final answers despite grounding on irrelevant regions, and (2) they may successfully locate the correct evidence but still fail to utilize it to reach accurate conclusions. Our findings demonstrate that ViEBench can serve as a more explainable and practical benchmark for comprehensively evaluating the effectiveness agentic VLMs. The codes will be released at: https://github.com/Xuchen-Li/ViEBench.