NICE FACT: Diagnosing and Calibrating VLMs in Quantitative Reasoning for Kinematic Physics

TL;DR

This paper introduces NICE and FACT, a diagnostic framework for evaluating and calibrating vision-language models' understanding of physical reasoning and confidence reliability in kinematic physics tasks.

Contribution

It presents a novel dual-diagnostic paradigm and calibration method to assess and improve VLMs' physical reasoning and confidence calibration.

Findings

Models often fail to identify visual preconditions.

Models do not reliably utilize physical laws for reasoning.

The diagnostic tools reveal significant gaps in model understanding.

Abstract

The ability to derive precise spatial and physical insights is a cornerstone of vision-language models (VLMs), yet their poor performances in related spatial intelligence tasks such as physical reasoning remain a fundamental barrier. The community critically lacks a scientific analysis revealing whether VLMs faithfully reach answers or plausibly make guesses. This work aims to provide a fundamental understanding of how VLMs perceive the physical world, and utilize physical laws, while assessing the reliability of model confidence. We propose NICE and FACT, a dual-diagnostic paradigm that explicitly decomposes quantitative reasoning for kinematic physics: FACT diagnoses visual fidelity, physical law comprehension, and temporal grounding. NICE studies our novel neighborhood-informed calibration method and novel metrics to evaluate and calibrate confidence reliability. Evaluated across 6 latest state-of-the-art VLMs, we uncover that models fail to identify visual preconditions or utilize necessary physical laws to reach answers. This work highlights and establishes a standardized diagnostic paradigm to guide the development of faithful, physically-grounded VLMs.