ELGAR: Expressive Cello Performance Motion Generation for Audio Rendition

TL;DR

ELGAR is a diffusion-based framework that generates expressive whole-body cello performance motions from audio, incorporating interactive contact losses and novel evaluation metrics, advancing realistic instrument motion synthesis.

Contribution

The paper introduces ELGAR, a novel diffusion model for audio-driven cello motion generation, with new contact loss functions and specialized metrics for performance evaluation.

Findings

ELGAR effectively generates realistic, fast, and complex cello motions.

The proposed contact losses improve interaction authenticity.

New metrics correlate well with motion-audio semantic alignment.

Abstract

The art of instrument performance stands as a vivid manifestation of human creativity and emotion. Nonetheless, generating instrument performance motions is a highly challenging task, as it requires not only capturing intricate movements but also reconstructing the complex dynamics of the performer-instrument interaction. While existing works primarily focus on modeling partial body motions, we propose Expressive ceLlo performance motion Generation for Audio Rendition (ELGAR), a state-of-the-art diffusion-based framework for whole-body fine-grained instrument performance motion generation solely from audio. To emphasize the interactive nature of the instrument performance, we introduce Hand Interactive Contact Loss (HICL) and Bow Interactive Contact Loss (BICL), which effectively guarantee the authenticity of the interplay. Moreover, to better evaluate whether the generated motions align with the semantic context of the music audio, we design novel metrics specifically for string instrument performance motion generation, including finger-contact distance, bow-string distance, and bowing score. Extensive evaluations and ablation studies are conducted to validate the efficacy of the proposed methods. In addition, we put forward a motion generation dataset SPD-GEN, collated and normalized from the MoCap dataset SPD. As demonstrated, ELGAR has shown great potential in generating instrument performance motions with complicated and fast interactions, which will promote further development in areas such as animation, music education, interactive art creation, etc.