VisPhyWorld: Probing Physical Reasoning via Code-Driven Video Reconstruction

TL;DR

VisPhyWorld introduces an execution-based framework for evaluating physical reasoning in models by generating and inspecting executable simulator code from visual data, enabling more testable and falsifiable assessments.

Contribution

The paper presents VisPhyWorld, a novel framework that evaluates physical reasoning through code generation, and introduces VisPhyBench, a comprehensive benchmark for physical scene understanding.

Findings

State-of-the-art models excel in semantic scene understanding but struggle with physical parameter inference.

The pipeline achieves 97.7% valid reconstructed videos before fallback.

Models have difficulty simulating consistent physical dynamics.

Abstract

Evaluating whether Multimodal Large Language Models (MLLMs) genuinely reason about physical dynamics remains challenging. Most existing benchmarks rely on recognition-style protocols such as Visual Question Answering (VQA) and Violation of Expectation (VoE), which can often be answered without committing to an explicit, testable physical hypothesis. We propose VisPhyWorld, an execution-based framework that evaluates physical reasoning by requiring models to generate executable simulator code from visual observations. By producing runnable code, the inferred world representation is directly inspectable, editable, and falsifiable. This separates physical reasoning from rendering. Building on this framework, we introduce VisPhyBench, comprising 209 evaluation scenes derived from 108 physical templates and a systematic protocol that evaluates how well models reconstruct appearance and reproduce physically plausible motion. Our pipeline produces valid reconstructed videos in 97.7% of benchmark runs before fallback. Experiments show that while state-of-the-art MLLMs achieve strong semantic scene understanding, they struggle to accurately infer physical parameters and to simulate consistent physical dynamics. Our code is available https://github.com/TIGER-AI-Lab/VisPhyWorld