MM-ALT: A Multimodal Automatic Lyric Transcription System

TL;DR

This paper introduces MM-ALT, a multimodal system for automatic lyric transcription that leverages audio, lip movement videos, and IMU data to improve accuracy and robustness against noise, supported by a new dataset.

Contribution

The paper presents a novel multimodal ALT system with a new dataset and a Residual Cross Attention mechanism for data fusion, advancing robustness and accuracy.

Findings

MM-ALT outperforms audio-only systems in noisy conditions

The Residual Cross Attention mechanism effectively fuses multimodal data

The new N20EM dataset enables multimodal ALT research

Abstract

Automatic lyric transcription (ALT) is a nascent field of study attracting increasing interest from both the speech and music information retrieval communities, given its significant application potential. However, ALT with audio data alone is a notoriously difficult task due to instrumental accompaniment and musical constraints resulting in degradation of both the phonetic cues and the intelligibility of sung lyrics. To tackle this challenge, we propose the MultiModal Automatic Lyric Transcription system (MM-ALT), together with a new dataset, N20EM, which consists of audio recordings, videos of lip movements, and inertial measurement unit (IMU) data of an earbud worn by the performing singer. We first adapt the wav2vec 2.0 framework from automatic speech recognition (ASR) to the ALT task. We then propose a video-based ALT method and an IMU-based voice activity detection (VAD) method. In addition, we put forward the Residual Cross Attention (RCA) mechanism to fuse data from the three modalities (i.e., audio, video, and IMU). Experiments show the effectiveness of our proposed MM-ALT system, especially in terms of noise robustness. Project page is at https://n20em.github.io.