When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought

TL;DR

MIRA is a benchmark that evaluates models' ability to generate and utilize intermediate visual images for reasoning, highlighting the importance of visual cues in complex problem-solving tasks.

Contribution

This paper introduces MIRA, a multimodal benchmark with a new evaluation protocol emphasizing intermediate visual reasoning, and demonstrates the limitations of current models relying solely on text.

Findings

Models improve performance with visual cues, with an average gain of 33.7%.

Existing models perform poorly with text-only prompts on visual reasoning tasks.

Visual-CoT prompts provide limited improvements over textual prompts alone.

Abstract

We propose MIRA, a new benchmark designed to evaluate models in scenarios where generating intermediate visual images is essential for successful reasoning. Unlike traditional CoT methods that rely solely on text, tasks in MIRA require models to generate and utilize intermediate images - such as sketches, structural diagrams, or path drawings - to guide their reasoning process. This setup closely mirrors how humans solve complex problems through "drawing to think". To solve this, MIRA focuses on tasks that are intrinsically challenging and involve complex structures, spatial relationships, or reasoning steps that are difficult to express through language alone. To ensure that our evaluation data is of high-quality, we include 546 multimodal problems, annotated with intermediate visual images and final answers. We also propose a unified evaluation protocol for MIRA that spans three levels of evaluation input: direct input with image and question only, text-only CoT input with image and thinking prompts, and Visual-CoT input with both annotated image clues and textual thinking prompts. To probe the upper bound of model capacity on our benchmark, we also report pass@k and majority voting accuracies under different k settings. Experimental results show that existing multimodal large language models, including strongest private models as well as strong open-weight models, perform poorly when relying solely on textual prompts. However, when intermediate visual cues are provided, model performance improves consistently, yielding an average relative gain of 33.7% across all models and tasks. We also probe the upper bound by expanding the search space and designing textual prompts aligned with Visual-CoT, but both yield only limited improvements compared to our Visual-CoT setting. These results underscore the critical role of imagined visual information in enabling successful reasoning on MIRA.