AMG: Avatar Motion Guided Video Generation

TL;DR

AMG is a novel method that combines 2D photorealism with 3D controllability for human video generation by conditioning diffusion models on 3D avatar renderings, enabling multi-person generation with precise control.

Contribution

It introduces a new approach that integrates 2D and 3D techniques for realistic, controllable human video synthesis, including a data pipeline for avatar reconstruction from videos.

Findings

Outperforms existing methods in realism and adaptability.

Enables multi-person video generation with precise control.

Supports control over camera, human motion, and background.

Abstract

Human video generation task has gained significant attention with the advancement of deep generative models. Generating realistic videos with human movements is challenging in nature, due to the intricacies of human body topology and sensitivity to visual artifacts. The extensively studied 2D media generation methods take advantage of massive human media datasets, but struggle with 3D-aware control; whereas 3D avatar-based approaches, while offering more freedom in control, lack photorealism and cannot be harmonized seamlessly with background scene. We propose AMG, a method that combines the 2D photorealism and 3D controllability by conditioning video diffusion models on controlled rendering of 3D avatars. We additionally introduce a novel data processing pipeline that reconstructs and renders human avatar movements from dynamic camera videos. AMG is the first method that enables multi-person diffusion video generation with precise control over camera positions, human motions, and background style. We also demonstrate through extensive evaluation that it outperforms existing human video generation methods conditioned on pose sequences or driving videos in terms of realism and adaptability.