Vid2Sim: Generalizable, Video-based Reconstruction of Appearance, Geometry and Physics for Mesh-free Simulation

TL;DR

Vid2Sim is a fast, generalizable framework that reconstructs physical properties and geometry from videos and enables efficient, mesh-free simulation without extensive scene-specific tuning.

Contribution

The paper introduces Vid2Sim, a novel mesh-free, video-based method that efficiently reconstructs physical properties and geometry with minimal optimization and high generalizability.

Findings

Achieves high accuracy in geometry and physical property reconstruction.

Operates with significantly reduced optimization time (a few minutes).

Enables efficient, high-quality mesh-free simulation post-reconstruction.

Abstract

Faithfully reconstructing textured shapes and physical properties from videos presents an intriguing yet challenging problem. Significant efforts have been dedicated to advancing such a system identification problem in this area. Previous methods often rely on heavy optimization pipelines with a differentiable simulator and renderer to estimate physical parameters. However, these approaches frequently necessitate extensive hyperparameter tuning for each scene and involve a costly optimization process, which limits both their practicality and generalizability. In this work, we propose a novel framework, Vid2Sim, a generalizable video-based approach for recovering geometry and physical properties through a mesh-free reduced simulation based on Linear Blend Skinning (LBS), offering high computational efficiency and versatile representation capability. Specifically, Vid2Sim first reconstructs the observed configuration of the physical system from video using a feed-forward neural network trained to capture physical world knowledge. A lightweight optimization pipeline then refines the estimated appearance, geometry, and physical properties to closely align with video observations within just a few minutes. Additionally, after the reconstruction, Vid2Sim enables high-quality, mesh-free simulation with high efficiency. Extensive experiments demonstrate that our method achieves superior accuracy and efficiency in reconstructing geometry and physical properties from video data.