Music Gesture for Visual Sound Separation

TL;DR

This paper introduces 'Music Gesture,' a keypoint-based visual representation that improves the separation of audio signals from visual music performances, especially when distinguishing similar instruments or duets.

Contribution

It proposes a novel keypoint-based structured representation and a graph network-based model to enhance audio-visual correlation for music sound separation.

Findings

Significant improvements on hetero-musical separation benchmarks

First effective homo-musical separation for duets of piano, flute, and trumpet

Demonstrated robustness across three music performance datasets

Abstract

Recent deep learning approaches have achieved impressive performance on visual sound separation tasks. However, these approaches are mostly built on appearance and optical flow like motion feature representations, which exhibit limited abilities to find the correlations between audio signals and visual points, especially when separating multiple instruments of the same types, such as multiple violins in a scene. To address this, we propose "Music Gesture," a keypoint-based structured representation to explicitly model the body and finger movements of musicians when they perform music. We first adopt a context-aware graph network to integrate visual semantic context with body dynamics, and then apply an audio-visual fusion model to associate body movements with the corresponding audio signals. Experimental results on three music performance datasets show: 1) strong improvements upon benchmark metrics for hetero-musical separation tasks (i.e. different instruments); 2) new ability for effective homo-musical separation for piano, flute, and trumpet duets, which to our best knowledge has never been achieved with alternative methods. Project page: http://music-gesture.csail.mit.edu.