PhysNote: Self-Knowledge Notes for Evolvable Physical Reasoning in Vision-Language Model

TL;DR

PhysNote enhances vision-language models' physical reasoning by externalizing and refining knowledge through self-generated notes, improving accuracy on dynamic real-world physics problems.

Contribution

Introduces PhysNote, a framework enabling VLMs to externalize, organize, and refine physical knowledge, addressing identity drift and volatility in reasoning.

Findings

Achieves 56.68% accuracy on PhysBench, outperforming baselines.

Improves physical reasoning across multiple domains.

Stabilizes perception with spatio-temporal canonicalization.

Abstract

Vision-Language Models (VLMs) have demonstrated strong performance on textbook-style physics problems, yet they frequently fail when confronted with dynamic real-world scenarios that require temporal consistency and causal reasoning across frames. We identify two fundamental challenges underlying these failures: (1) spatio-temporal identity drift, where objects lose their physical identity across successive frames and break causal chains, and (2) volatility of inference-time insights, where a model may occasionally produce correct physical reasoning but never consolidates it for future reuse. To address these challenges, we propose PhysNote, an agentic framework that enables VLMs to externalize and refine physical knowledge through self-generated "Knowledge Notes." PhysNote stabilizes dynamic perception through spatio-temporal canonicalization, organizes self-generated insights into a hierarchical knowledge repository, and drives an iterative reasoning loop that grounds hypotheses in visual evidence before consolidating verified knowledge. Experiments on PhysBench demonstrate that PhysNote achieves 56.68% overall accuracy, a 4.96% improvement over the best multi-agent baseline, with consistent gains across all four physical reasoning domains.