VioPose: Violin Performance 4D Pose Estimation by Hierarchical Audiovisual Inference

TL;DR

VioPose is a multimodal hierarchical network that estimates precise 4D violinist poses from monocular videos by leveraging audiovisual cues, outperforming existing methods and supported by a large, diverse dataset.

Contribution

The paper introduces VioPose, a novel audiovisual hierarchical model for 4D pose estimation that addresses limitations of visual-only methods in capturing subtle musical motions.

Findings

Outperforms state-of-the-art visual pose estimation methods

Accurately captures fast and subtle violin movements

Provides a large, diverse dataset for violin performance analysis

Abstract

Musicians delicately control their bodies to generate music. Sometimes, their motions are too subtle to be captured by the human eye. To analyze how they move to produce the music, we need to estimate precise 4D human pose (3D pose over time). However, current state-of-the-art (SoTA) visual pose estimation algorithms struggle to produce accurate monocular 4D poses because of occlusions, partial views, and human-object interactions. They are limited by the viewing angle, pixel density, and sampling rate of the cameras and fail to estimate fast and subtle movements, such as in the musical effect of vibrato. We leverage the direct causal relationship between the music produced and the human motions creating them to address these challenges. We propose VioPose: a novel multimodal network that hierarchically estimates dynamics. High-level features are cascaded to low-level features and integrated into Bayesian updates. Our architecture is shown to produce accurate pose sequences, facilitating precise motion analysis, and outperforms SoTA. As part of this work, we collected the largest and the most diverse calibrated violin-playing dataset, including video, sound, and 3D motion capture poses. Code and dataset can be found in our project page \url{https://sj-yoo.info/viopose/}.