SyncViolinist: Music-Oriented Violin Motion Generation Based on Bowing and Fingering
Hiroki Nishizawa, Keitaro Tanaka, Asuka Hirata, Shugo Yamaguchi, Qi, Feng, Masatoshi Hamanaka, Shigeo Morishima

TL;DR
SyncViolinist is an end-to-end framework that generates realistic, synchronized violin motion from audio alone, capturing detailed bowing and fingering movements for enhanced digital media production.
Contribution
It introduces a novel multi-stage approach that extracts detailed playing information from audio to produce accurate violin performance motions without additional input data.
Findings
Outperforms state-of-the-art methods in qualitative and quantitative evaluations.
Successfully generates synchronized violin motions from unseen audio.
Validated by professional violinist subjective assessments.
Abstract
Automatically generating realistic musical performance motion can greatly enhance digital media production, often involving collaboration between professionals and musicians. However, capturing the intricate body, hand, and finger movements required for accurate musical performances is challenging. Existing methods often fall short due to the complex mapping between audio and motion, typically requiring additional inputs like scores or MIDI data. In this work, we present SyncViolinist, a multi-stage end-to-end framework that generates synchronized violin performance motion solely from audio input. Our method overcomes the challenge of capturing both global and fine-grained performance features through two key modules: a bowing/fingering module and a motion generation module. The bowing/fingering module extracts detailed playing information from the audio, which the motion generation…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMusic Technology and Sound Studies · Music and Audio Processing · Human Motion and Animation
