SyncViolinist: Music-Oriented Violin Motion Generation Based on Bowing and Fingering

TL;DR

SyncViolinist is an end-to-end framework that generates realistic, synchronized violin motion from audio alone, capturing detailed bowing and fingering movements for enhanced digital media production.

Contribution

It introduces a novel multi-stage approach that extracts detailed playing information from audio to produce accurate violin performance motions without additional input data.

Findings

Outperforms state-of-the-art methods in qualitative and quantitative evaluations.

Successfully generates synchronized violin motions from unseen audio.

Validated by professional violinist subjective assessments.

Abstract

Automatically generating realistic musical performance motion can greatly enhance digital media production, often involving collaboration between professionals and musicians. However, capturing the intricate body, hand, and finger movements required for accurate musical performances is challenging. Existing methods often fall short due to the complex mapping between audio and motion, typically requiring additional inputs like scores or MIDI data. In this work, we present SyncViolinist, a multi-stage end-to-end framework that generates synchronized violin performance motion solely from audio input. Our method overcomes the challenge of capturing both global and fine-grained performance features through two key modules: a bowing/fingering module and a motion generation module. The bowing/fingering module extracts detailed playing information from the audio, which the motion generation module uses to create precise, coordinated body motions reflecting the temporal granularity and nature of the violin performance. We demonstrate the effectiveness of SyncViolinist with significantly improved qualitative and quantitative results from unseen violin performance audio, outperforming state-of-the-art methods. Extensive subjective evaluations involving professional violinists further validate our approach. The code and dataset are available at https://github.com/Kakanat/SyncViolinist.