VisChainBench: A Benchmark for Multi-Turn, Multi-Image Visual Reasoning Beyond Language Priors

TL;DR

VisChainBench introduces a comprehensive benchmark to evaluate large vision-language models' ability to perform multi-step, context-dependent visual reasoning across diverse, sequential tasks with minimal language cues.

Contribution

The paper presents VisChainBench, a large-scale, multi-domain benchmark designed to assess multi-turn visual reasoning beyond language priors, created with a multi-agent pipeline for diversity and bias control.

Findings

LVLMs show limited multi-step reasoning capabilities.

Benchmark reveals gaps in current models' visual reasoning skills.

VisChainBench sets a new standard for multi-turn visual reasoning evaluation.

Abstract

Understanding multi-image, multi-turn scenarios is a critical yet underexplored capability for Large Vision-Language Models (LVLMs). Existing benchmarks predominantly focus on static or horizontal comparisons -- e.g., spotting visual differences or assessing appropriateness -- while relying heavily on language cues. Such settings overlook progressive, context-dependent reasoning and the challenge of visual-to-visual inference. To bridge this gap, we present VisChainBench, a large-scale benchmark designed to rigorously evaluate LVLMs' ability to perform multi-step visual reasoning across sequential, interdependent tasks with minimal language guidance. VisChainBench contains 1,457 tasks spanning over 20,000 images across three diverse domains (e.g., daily scenarios, engineering troubleshooting), structured to mimic real-world decision-making processes. Uniquely, the benchmark is constructed using a multi-agent generation pipeline, ensuring high visual diversity and controlled language bias. All the benchmark data and code for benchmark construction are available for viewing and download via following Link: https://huggingface.co/datasets/eyehole/VisChainBench