Deep Audio-Visual Singing Voice Transcription based on Self-Supervised Learning Models

TL;DR

This paper introduces a multimodal, self-supervised learning approach for singing voice transcription that leverages audio and video data, significantly improving noise robustness and reducing the need for annotated data.

Contribution

It presents a novel multimodal dataset and adapts self-supervised models from speech to singing voice transcription, enhancing performance and generalization.

Findings

Audio-only system outperforms state-of-the-art methods.

Video-only system achieves about 80% accuracy in note detection.

Audio-visual system significantly improves noise robustness.

Abstract

Singing voice transcription converts recorded singing audio to musical notation. Sound contamination (such as accompaniment) and lack of annotated data make singing voice transcription an extremely difficult task. We take two approaches to tackle the above challenges: 1) introducing multimodal learning for singing voice transcription together with a new multimodal singing dataset, N20EMv2, enhancing noise robustness by utilizing video information (lip movements to predict the onset/offset of notes), and 2) adapting self-supervised learning models from the speech domain to the singing voice transcription task, significantly reducing annotated data requirements while preserving pretrained features. We build a self-supervised learning based audio-only singing voice transcription system, which not only outperforms current state-of-the-art technologies as a strong baseline, but also generalizes well to out-of-domain singing data. We then develop a self-supervised learning based video-only singing voice transcription system that detects note onsets and offsets with an accuracy of about 80\%. Finally, based on the powerful acoustic and visual representations extracted by the above two systems as well as the feature fusion design, we create an audio-visual singing voice transcription system that improves the noise robustness significantly under different acoustic environments compared to the audio-only systems.