WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actions

TL;DR

WonderPlay is a framework that combines physics simulation and video generation to create realistic, action-conditioned 3D scenes from a single image, enabling diverse dynamic content interaction.

Contribution

It introduces a hybrid generative simulator that integrates physics-based simulation with diffusion-based video generation for diverse 3D scene synthesis.

Findings

Enables interaction with scenes of cloth, sand, snow, liquid, smoke, elastic, and rigid bodies.

Achieves realistic motion through a closed-loop between physics simulation and video generation.

Supports user control with diverse dynamic scene content.

Abstract

WonderPlay is a novel framework integrating physics simulation with video generation for generating action-conditioned dynamic 3D scenes from a single image. While prior works are restricted to rigid body or simple elastic dynamics, WonderPlay features a hybrid generative simulator to synthesize a wide range of 3D dynamics. The hybrid generative simulator first uses a physics solver to simulate coarse 3D dynamics, which subsequently conditions a video generator to produce a video with finer, more realistic motion. The generated video is then used to update the simulated dynamic 3D scene, closing the loop between the physics solver and the video generator. This approach enables intuitive user control to be combined with the accurate dynamics of physics-based simulators and the expressivity of diffusion-based video generators. Experimental results demonstrate that WonderPlay enables users to interact with various scenes of diverse content, including cloth, sand, snow, liquid, smoke, elastic, and rigid bodies -- all using a single image input. Code will be made public. Project website: https://kyleleey.github.io/WonderPlay/