ISO-Bench: Benchmarking Multimodal Causal Reasoning in Visual-Language Models through Procedural Plans

TL;DR

ISO-Bench is a new benchmark designed to evaluate multimodal models' ability to understand causal relationships between visual observations and procedural text, revealing significant gaps in current models' reasoning capabilities.

Contribution

The paper introduces ISO-Bench, a novel benchmark for assessing causal reasoning in multimodal models, and provides an analysis of current models' performance and limitations.

Findings

Current models achieve low zero-shot F1 scores (max 0.57).

Chain-of-thought reasoning offers only modest improvements.

Models lag far behind human performance (0.98 F1).

Abstract

Understanding causal relationships across modalities is a core challenge for multimodal models operating in real-world environments. We introduce ISO-Bench, a benchmark for evaluating whether models can infer causal dependencies between visual observations and procedural text. Each example presents an image of a task step and a text snippet from a plan, with the goal of deciding whether the visual step occurs before or after the referenced text step. Evaluation results on ten frontier vision-language models show underwhelming performance: the best zero-shot F1 is only 0.57, and chain-of-thought reasoning yields only modest gains (up to 0.62 F1), largely behind humans (0.98 F1). Our analysis further highlights concrete directions for improving causal understanding in multimodal models.