DreamDance: Animating Character Art via Inpainting Stable Gaussian Worlds

TL;DR

DreamDance introduces a two-step inpainting framework for animating character art with stable, camera-aware scene rendering and pose-aware character integration, resulting in high-quality, consistent animations.

Contribution

It reformulates character animation as camera-aware scene inpainting and pose-aware video inpainting, leveraging large-scale Gaussian fields and a DiT-based model for improved stability and quality.

Findings

Produces high-quality, consistent character animations

Effectively handles complex camera trajectories

Demonstrates generalizability across diverse scenes

Abstract

This paper presents DreamDance, a novel character art animation framework capable of producing stable, consistent character and scene motion conditioned on precise camera trajectories. To achieve this, we re-formulate the animation task as two inpainting-based steps: Camera-aware Scene Inpainting and Pose-aware Video Inpainting. The first step leverages a pre-trained image inpainting model to generate multi-view scene images from the reference art and optimizes a stable large-scale Gaussian field, which enables coarse background video rendering with camera trajectories. However, the rendered video is rough and only conveys scene motion. To resolve this, the second step trains a pose-aware video inpainting model that injects the dynamic character into the scene video while enhancing background quality. Specifically, this model is a DiT-based video generation model with a gating strategy that adaptively integrates the character's appearance and pose information into the base background video. Through extensive experiments, we demonstrate the effectiveness and generalizability of DreamDance, producing high-quality and consistent character animations with remarkable camera dynamics.