Vision Language Models Cannot Reason About Physical Transformation

TL;DR

This paper evaluates whether current Vision Language Models truly understand physical transformations, finding they largely fail to maintain invariant representations of physical properties across dynamic scenes, despite some superficial improvements.

Contribution

The study introduces ConservationBench, a new benchmark for assessing physical invariance in VLMs, revealing systematic failures in their reasoning about physical transformations.

Findings

Models perform near chance on conservation tasks.

Textual priors favor invariance, but visual content reduces performance.

Standard techniques do not improve models' understanding of physical transformations.

Abstract

Understanding physical transformations is fundamental for reasoning in dynamic environments. While Vision Language Models (VLMs) show promise in embodied applications, whether they genuinely understand physical transformations remains unclear. We introduce ConservationBench evaluating conservation -- whether physical quantities remain invariant under transformations. Spanning four properties with paired conserving/non-conserving scenarios, we generate 23,040 questions across 112 VLMs. Results reveal systematic failure: performance remains near chance with improvements on conservation tasks accompanied by drops on controls. Control experiments show strong textual priors favoring invariance, yet models perform worse with visual content. Neither temporal resolution, prompting, nor curated sampling helps. These findings show that current VLMs fail to maintain transformation-invariant representations of physical properties across dynamic scenes.