Unsupervised Generative Adversarial Alignment Representation for Sheet music, Audio and Lyrics

TL;DR

This paper introduces an unsupervised adversarial model that learns shared deep representations across sheet music, audio, and lyrics, enabling improved alignment and transfer between these musical modalities.

Contribution

The novel UGAAR model jointly trains three neural network branches to align sheet music, audio, and lyrics without supervision, leveraging adversarial learning and shared latent space.

Findings

Demonstrates effective alignment among three modalities

Enables transfer of relationships from audio-sheet to lyrics-others

Shows feasibility of unsupervised deep multimodal representation learning

Abstract

Sheet music, audio, and lyrics are three main modalities during writing a song. In this paper, we propose an unsupervised generative adversarial alignment representation (UGAAR) model to learn deep discriminative representations shared across three major musical modalities: sheet music, lyrics, and audio, where a deep neural network based architecture on three branches is jointly trained. In particular, the proposed model can transfer the strong relationship between audio and sheet music to audio-lyrics and sheet-lyrics pairs by learning the correlation in the latent shared subspace. We apply CCA components of audio and sheet music to establish new ground truth. The generative (G) model learns the correlation of two couples of transferred pairs to generate new audio-sheet pair for a fixed lyrics to challenge the discriminative (D) model. The discriminative model aims at distinguishing the input which is from the generative model or the ground truth. The two models simultaneously train in an adversarial way to enhance the ability of deep alignment representation learning. Our experimental results demonstrate the feasibility of our proposed UGAAR for alignment representation learning among sheet music, audio, and lyrics.