Order from Chaos: Physical World Understanding from Glitchy Gameplay Videos
Meng Cao, Haoran Tang, Haoze Zhao, Mingfei Han, Ruyang Liu, Qiang Sun, Xiaojun Chang, Ian Reid, Xiaodan Liang

TL;DR
This paper introduces PhysGame, a large dataset of glitch-centric QA pairs from gameplay videos, to improve AI understanding of physical principles by leveraging visual anomalies as supervision.
Contribution
It presents a novel glitch-based supervision paradigm, a comprehensive dataset, and a benchmark for physical reasoning, enhancing AI's physical world understanding from gameplay videos.
Findings
PhysGame improves physical reasoning transferability to real-world tasks.
Models fine-tuned on PhysGame outperform baselines in glitch detection.
Learning from gameplay anomalies enhances robustness and generalization.
Abstract
Understanding the physical world, including object dynamics, material properties, and causal interactions, remains a core challenge in artificial intelligence. Although recent multi-modal large language models (MLLMs) have demonstrated impressive general reasoning capabilities, they still fall short of achieving human-level understanding of physical principles. Existing datasets for physical reasoning either rely on real-world videos, which incur high annotation costs, or on synthetic simulations, which suffer from limited realism and diversity. In this paper, we propose a novel paradigm that leverages glitches in gameplay videos, referring to visual anomalies that violate predefined physical laws, as a rich and scalable supervision source for physical world understanding. We introduce PhysGame, an meta information guided instruction-tuning dataset containing 140,057 glitch-centric…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Artificial Intelligence in Games · Topic Modeling
