Towards Physical Understanding in Video Generation: A 3D Point Regularization Approach

TL;DR

This paper introduces a 3D-aware video generation framework that enhances the physical realism of generated videos by integrating 3D geometry, regularizing shape and motion, and improving handling of contact-rich scenarios.

Contribution

It presents a novel 3D point regularization approach and a new dataset, PointVid, to improve the physical plausibility of video generation models.

Findings

Enhanced video quality with reduced artifacts.

Improved handling of contact-rich and shape-aware scenarios.

Seamless integration with existing diffusion models.

Abstract

We present a novel video generation framework that integrates 3-dimensional geometry and dynamic awareness. To achieve this, we augment 2D videos with 3D point trajectories and align them in pixel space. The resulting 3D-aware video dataset, PointVid, is then used to fine-tune a latent diffusion model, enabling it to track 2D objects with 3D Cartesian coordinates. Building on this, we regularize the shape and motion of objects in the video to eliminate undesired artifacts, e.g., non-physical deformation. Consequently, we enhance the quality of generated RGB videos and alleviate common issues like object morphing, which are prevalent in current video models due to a lack of shape awareness. With our 3D augmentation and regularization, our model is capable of handling contact-rich scenarios such as task-oriented videos, where 3D information is essential for perceiving shape and motion of interacting solids. Our method can be seamlessly integrated into existing video diffusion models to improve their visual plausibility.