Dance Your Latents: Consistent Dance Generation through Spatial-temporal Subspace Attention Guided by Motion Flow

TL;DR

This paper introduces Dance-Your-Latents, a novel framework that enhances the spatiotemporal consistency of dance video generation by using spatial-temporal subspace attention guided by motion flow, reducing artifacts like flickering.

Contribution

The paper proposes a new spatial-temporal subspace attention mechanism and motion flow guided alignment to improve dance video coherence in generative models.

Findings

Significantly improves spatiotemporal consistency in generated dance videos

Reduces artifacts such as ghosting and flickering

Demonstrates effectiveness on TikTok dataset

Abstract

The advancement of generative AI has extended to the realm of Human Dance Generation, demonstrating superior generative capacities. However, current methods still exhibit deficiencies in achieving spatiotemporal consistency, resulting in artifacts like ghosting, flickering, and incoherent motions. In this paper, we present Dance-Your-Latents, a framework that makes latents dance coherently following motion flow to generate consistent dance videos. Firstly, considering that each constituent element moves within a confined space, we introduce spatial-temporal subspace-attention blocks that decompose the global space into a combination of regular subspaces and efficiently model the spatiotemporal consistency within these subspaces. This module enables each patch pay attention to adjacent areas, mitigating the excessive dispersion of long-range attention. Furthermore, observing that body part's movement is guided by pose control, we design motion flow guided subspace align & restore. This method enables the attention to be computed on the irregular subspace along the motion flow. Experimental results in TikTok dataset demonstrate that our approach significantly enhances spatiotemporal consistency of the generated videos.