Harmony-Aware Music-driven Motion Synthesis with Perceptual Constraint on UGC Datasets

TL;DR

This paper introduces a harmony-aware GAN framework for music-driven 3D human motion synthesis from UGC datasets, emphasizing perceptual harmony constraints to improve rhythmic synchronization and realism.

Contribution

It proposes a novel harmony evaluation strategy with beat detection and saliency weighting, integrated into a GAN to enhance rhythmic harmony in generated motions.

Findings

Outperforms existing methods in rhythmic harmony metrics

Achieves realistic and synchronized motions with limited training data

Demonstrates effectiveness of perceptual harmony constraints in motion synthesis

Abstract

With the popularity of video-based user-generated content (UGC) on social media, harmony, as dictated by human perceptual principles, is critical in assessing the rhythmic consistency of audio-visual UGCs for better user engagement. In this work, we propose a novel harmony-aware GAN framework, following a specifically designed harmony evaluation strategy to enhance rhythmic synchronization in the automatic music-to-motion synthesis using a UGC dance dataset. This harmony strategy utilizes refined cross-modal beat detection to capture closely correlated audio and visual rhythms in an audio-visual pair. To mimic human attention mechanism, we introduce saliency-based beat weighting and interval-driven beat alignment, which ensures accurate harmony score estimation consistent with human perception. Building on this strategy, our model, employing efficient encoder-decoder and depth-lifting designs, is adversarially trained based on categorized musical meter segments to generate realistic and rhythmic 3D human motions. We further incorporate our harmony evaluation strategy as a weakly supervised perceptual constraint to flexibly guide the synchronized audio-visual rhythms during the generation process. Experimental results show that our proposed model significantly outperforms other leading music-to-motion methods in rhythmic harmony, both quantitatively and qualitatively, even with limited UGC training data. Live samples 15 can be watched at: https://youtu.be/tWwz7yq4aUs