Video Creation by Demonstration
Yihong Sun, Hao Zhou, Liangzhe Yuan, Jennifer J. Sun, Yandong Li,, Xuhui Jia, Hartwig Adam, Bharath Hariharan, Long Zhao, Ting Liu

TL;DR
This paper introduces a novel method called $ ext{delta}$-Diffusion for generating realistic continuation videos from a demonstration video and a context image, enabling flexible and expressive video creation.
Contribution
It presents $ ext{delta}$-Diffusion, a self-supervised training approach that learns from unlabeled videos and uses implicit latent control for flexible video generation.
Findings
$ ext{delta}$-Diffusion outperforms baselines in human preference and machine evaluations.
The method enables interactive world simulation.
It effectively extracts action latents with minimal appearance leakage.
Abstract
We explore a novel video creation experience, namely Video Creation by Demonstration. Given a demonstration video and a context image from a different scene, we generate a physically plausible video that continues naturally from the context image and carries out the action concepts from the demonstration. To enable this capability, we present -Diffusion, a self-supervised training approach that learns from unlabeled videos by conditional future frame prediction. Unlike most existing video generation controls that are based on explicit signals, we adopts the form of implicit latent control for maximal flexibility and expressiveness required by general videos. By leveraging a video foundation model with an appearance bottleneck design on top, we extract action latents from demonstration videos for conditioning the generation process with minimal appearance leakage. Empirically,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGenerative Adversarial Networks and Image Synthesis · Human Motion and Animation · Cinema and Media Studies
