NICE FACT: Diagnosing and Calibrating VLMs in Quantitative Reasoning for Kinematic Physics
Jian Lan, Zhicheng Liu, Xinpeng Wang, Yuhao Zhou, Haokun Chen, Jiancheng Lv, Barbara Plank, Thomas Seidl

TL;DR
This paper introduces NICE and FACT, a diagnostic framework for evaluating and calibrating vision-language models' understanding of physical reasoning and confidence reliability in kinematic physics tasks.
Contribution
It presents a novel dual-diagnostic paradigm and calibration method to assess and improve VLMs' physical reasoning and confidence calibration.
Findings
Models often fail to identify visual preconditions.
Models do not reliably utilize physical laws for reasoning.
The diagnostic tools reveal significant gaps in model understanding.
Abstract
The ability to derive precise spatial and physical insights is a cornerstone of vision-language models (VLMs), yet their poor performances in related spatial intelligence tasks such as physical reasoning remain a fundamental barrier. The community critically lacks a scientific analysis revealing whether VLMs faithfully reach answers or plausibly make guesses. This work aims to provide a fundamental understanding of how VLMs perceive the physical world, and utilize physical laws, while assessing the reliability of model confidence. We propose NICE and FACT, a dual-diagnostic paradigm that explicitly decomposes quantitative reasoning for kinematic physics: FACT diagnoses visual fidelity, physical law comprehension, and temporal grounding. NICE studies our novel neighborhood-informed calibration method and novel metrics to evaluate and calibrate confidence reliability. Evaluated across 6…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
