QuantiPhy: A Quantitative Benchmark Evaluating Physical Reasoning Abilities of Vision-Language Models

TL;DR

QuantiPhy is a new benchmark that quantitatively assesses vision-language models' ability to reason about physical properties like size, velocity, and acceleration from videos, revealing gaps between plausibility and numerical accuracy.

Contribution

This paper introduces QuantiPhy, the first benchmark for quantitatively evaluating physical reasoning in vision-language models using video-text data with ground truth measurements.

Findings

State-of-the-art VLMs show a gap between qualitative plausibility and numerical correctness.

Models tend to rely on pre-trained world knowledge rather than visual inputs.

QuantiPhy provides a scalable, rigorous testbed for physical reasoning evaluation.

Abstract

Understanding the physical world is essential for generalist AI agents. However, it remains unclear whether state-of-the-art vision perception models (e.g., large VLMs) can reason physical properties quantitatively. Existing evaluations are predominantly VQA-based and qualitative, offering limited insight into whether these models can infer the kinematic quantities of moving objects from video observations. To address this, we present QuantiPhy, the first benchmark designed to quantitatively measure a VLM's physical reasoning ability. Comprising more than 3.3K video-text instances with numerical ground truth, QuantiPhy evaluates a VLM's performance on estimating an object's size, velocity, and acceleration at a given timestamp, using one of these properties as an input prior. The benchmark standardizes prompts and scoring to assess numerical accuracy, enabling fair comparisons across models. Our experiments on state-of-the-art VLMs reveal a consistent gap between their qualitative plausibility and actual numerical correctness. We further provide an in-depth analysis of key factors like background noise, counterfactual priors, and strategic prompting and find that state-of-the-art VLMs lean heavily on pre-trained world knowledge rather than faithfully using the provided visual and textual inputs as references when reasoning kinematic properties quantitatively. QuantiPhy offers the first rigorous, scalable testbed to move VLMs beyond mere verbal plausibility toward a numerically grounded physical understanding.