MORSE-500: A Programmatically Controllable Video Benchmark to Stress-Test Multimodal Reasoning

TL;DR

MORSE-500 is a controllable, script-generated video benchmark designed to evaluate and stress-test multimodal reasoning across diverse, complex, and evolving scenarios, revealing significant gaps in current models.

Contribution

The paper introduces MORSE-500, a novel, programmatically generated video benchmark with adjustable difficulty, covering multiple reasoning categories to better evaluate multimodal intelligence.

Findings

State-of-the-art models perform poorly on MORSE-500, especially in abstract and planning tasks.

The benchmark's controllable generation pipeline allows systematic difficulty scaling.

MORSE-500 reveals substantial performance gaps in current multimodal reasoning systems.

Abstract

Despite rapid advances in vision-language models (VLMs), current benchmarks for multimodal reasoning fall short in three key dimensions. First, they overwhelmingly rely on static images, failing to capture the temporal complexity of real-world environments. Second, they narrowly focus on mathematical problem-solving, neglecting the broader spectrum of reasoning skills -- including abstract, physical, planning, spatial, and temporal capabilities -- required for robust multimodal intelligence. Third, many benchmarks quickly saturate, offering limited headroom for diagnosing failure modes or measuring continued progress. We introduce MORSE-500 (Multimodal Reasoning Stress-test Environment), a video benchmark composed of 500 fully scripted clips with embedded questions spanning six complementary reasoning categories. Each instance is programmatically generated using deterministic Python scripts (via Manim, Matplotlib, MoviePy), generative video models, and curated real footage. This script-driven design allows fine-grained control over visual complexity, distractor density, and temporal dynamics -- enabling difficulty to be scaled systematically as models improve. Unlike static benchmarks that become obsolete once saturated, MORSE-500 is built to evolve: its controllable generation pipeline supports the creation of arbitrarily challenging new instances, making it ideally suited for stress-testing next-generation models. Initial experiments with state-of-the-art systems -- including various Gemini 2.5 Pro and OpenAI o3 which represent the strongest available at the time, alongside strong open-source models -- reveal substantial performance gaps across all categories, with particularly large deficits in abstract and planning tasks. We release the full dataset, generation scripts, and evaluation harness to support transparent, reproducible, and forward-looking multimodal reasoning research.