MOSPA: Human Motion Generation Driven by Spatial Audio

TL;DR

This paper introduces MOSPA, a diffusion-based framework that generates realistic human motions driven by spatial audio, supported by a new comprehensive dataset, advancing the modeling of human responses to auditory stimuli.

Contribution

The paper presents the first spatial audio-driven human motion dataset and a diffusion-based model for realistic motion generation conditioned on spatial audio inputs.

Findings

MOSPA achieves state-of-the-art performance on spatial audio-driven motion generation.

The dataset enables diverse and high-quality motion-audio research.

The model effectively captures the relationship between spatial audio cues and human motion.

Abstract

Enabling virtual humans to dynamically and realistically respond to diverse auditory stimuli remains a key challenge in character animation, demanding the integration of perceptual modeling and motion synthesis. Despite its significance, this task remains largely unexplored. Most previous works have primarily focused on mapping modalities like speech, audio, and music to generate human motion. As of yet, these models typically overlook the impact of spatial features encoded in spatial audio signals on human motion. To bridge this gap and enable high-quality modeling of human movements in response to spatial audio, we introduce the first comprehensive Spatial Audio-Driven Human Motion (SAM) dataset, which contains diverse and high-quality spatial audio and motion data. For benchmarking, we develop a simple yet effective diffusion-based generative framework for human MOtion generation driven by SPatial Audio, termed MOSPA, which faithfully captures the relationship between body motion and spatial audio through an effective fusion mechanism. Once trained, MOSPA can generate diverse, realistic human motions conditioned on varying spatial audio inputs. We perform a thorough investigation of the proposed dataset and conduct extensive experiments for benchmarking, where our method achieves state-of-the-art performance on this task. Our code and model are publicly available at https://github.com/xsy27/Mospa-Acoustic-driven-Motion-Generation