Seeing the Wind from a Falling Leaf
Zhiyuan Gao, Jiageng Mao, Hong-Xing Yu, Haozhe Lou, Emily Yue-Ting Jia, Jernej Barbic, Jiajun Wu, Yue Wang

TL;DR
This paper introduces an end-to-end differentiable inverse graphics framework that infers invisible physical forces, like wind, from videos by modeling object geometry, physical properties, and interactions, enabling applications in physics-based video generation.
Contribution
The paper presents a novel approach to recover invisible forces from videos using a differentiable inverse graphics model, bridging vision and physics.
Findings
Successfully infers plausible force fields from videos.
Validates method on synthetic and real-world data.
Enables physics-based video editing and generation.
Abstract
A longstanding goal in computer vision is to model motions from videos, while the representations behind motions, i.e. the invisible physical interactions that cause objects to deform and move, remain largely unexplored. In this paper, we study how to recover the invisible forces from visual observations, e.g., estimating the wind field by observing a leaf falling to the ground. Our key innovation is an end-to-end differentiable inverse graphics framework, which jointly models object geometry, physical properties, and interactions directly from videos. Through backpropagation, our approach enables the recovery of force representations from object motions. We validate our method on both synthetic and real-world scenarios, and the results demonstrate its ability to infer plausible force fields from videos. Furthermore, we show the potential applications of our approach, including…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsHuman Pose and Action Recognition · 3D Shape Modeling and Analysis · Robot Manipulation and Learning
