DanceCamAnimator: Keyframe-Based Controllable 3D Dance Camera Synthesis

TL;DR

DanceCamAnimator is a novel framework that synthesizes controllable 3D dance camera movements from music and dance by using a keyframe-based approach, improving smoothness and controllability over previous methods.

Contribution

It introduces a three-stage process and an end-to-end framework for keyframe-based dance camera synthesis, inspired by human animation procedures.

Findings

Outperforms previous baselines quantitatively.

Produces smoother camera movements with reduced jitter.

Demonstrates powerful controllability with variable keyframe lengths.

Abstract

Synthesizing camera movements from music and dance is highly challenging due to the contradicting requirements and complexities of dance cinematography. Unlike human movements, which are always continuous, dance camera movements involve both continuous sequences of variable lengths and sudden drastic changes to simulate the switching of multiple cameras. However, in previous works, every camera frame is equally treated and this causes jittering and unavoidable smoothing in post-processing. To solve these problems, we propose to integrate animator dance cinematography knowledge by formulating this task as a three-stage process: keyframe detection, keyframe synthesis, and tween function prediction. Following this formulation, we design a novel end-to-end dance camera synthesis framework \textbf{DanceCamAnimator}, which imitates human animation procedures and shows powerful keyframe-based controllability with variable lengths. Extensive experiments on the DCM dataset demonstrate that our method surpasses previous baselines quantitatively and qualitatively. Code will be available at \url{https://github.com/Carmenw1203/DanceCamAnimator-Official}.