Skeleton2Stage: Reward-Guided Fine-Tuning for Physically Plausible Dance Generation

TL;DR

This paper introduces Skeleton2Stage, a reinforcement learning approach that fine-tunes dance generation models to produce physically plausible motions by incorporating physics-based rewards and addressing issues like freezing motions.

Contribution

It proposes a novel reward-guided fine-tuning method that improves physical plausibility in dance synthesis, incorporating anti-freezing rewards to preserve motion dynamics.

Findings

Significantly reduces body self-penetration and foot-ground contact anomalies.

Produces more realistic and aesthetically pleasing dance motions.

Enhances the physical plausibility of generated dances across multiple datasets.

Abstract

Despite advances in dance generation, most methods are trained in the skeletal domain and ignore mesh-level physical constraints. As a result, motions that look plausible as joint trajectories often exhibit body self-penetration and Foot-Ground Contact (FGC) anomalies when visualized with a human body mesh, reducing the aesthetic appeal of generated dances and limiting their real-world applications. We address this skeleton-to-mesh gap by deriving physics-based rewards from the body mesh and applying Reinforcement Learning Fine-Tuning (RLFT) to steer the diffusion model toward physically plausible motion synthesis under mesh visualization. Our reward design combines (i) an imitation reward that measures a motion's general plausibility by its imitability in a physical simulator (penalizing penetration and foot skating), and (ii) a Foot-Ground Deviation (FGD) reward with test-time FGD guidance to better capture the dynamic foot-ground interaction in dance. However, we find that the physics-based rewards tend to push the model to generate freezing motions for fewer physical anomalies and better imitability. To mitigate it, we propose an anti-freezing reward to preserve motion dynamics while maintaining physical plausibility. Experiments on multiple dance datasets consistently demonstrate that our method can significantly improve the physical plausibility of generated motions, yielding more realistic and aesthetically pleasing dances. The project page is available at: https://jjd1123.github.io/Skeleton2Stage/