PiCoGen2: Piano cover generation with transfer learning approach and weakly aligned data

TL;DR

PiCoGen2 introduces a transfer learning approach for piano cover generation that leverages weakly aligned data and high-level features to improve quality across multiple pop genres.

Contribution

The paper presents a novel transfer learning method using weakly aligned data and high-level feature extraction to enhance piano cover generation.

Findings

Outperforms baseline models on objective metrics

Achieves high-quality results across five pop genres

Utilizes weakly aligned data effectively

Abstract

Piano cover generation aims to create a piano cover from a pop song. Existing approaches mainly employ supervised learning and the training demands strongly-aligned and paired song-to-piano data, which is built by remapping piano notes to song audio. This would, however, result in the loss of piano information and accordingly cause inconsistencies between the original and remapped piano versions. To overcome this limitation, we propose a transfer learning approach that pre-trains our model on piano-only data and fine-tunes it on weakly-aligned paired data constructed without note remapping. During pre-training, to guide the model to learn piano composition concepts instead of merely transcribing audio, we use an existing lead sheet transcription model as the encoder to extract high-level features from the piano recordings. The pre-trained model is then fine-tuned on the paired song-piano data to transfer the learned composition knowledge to the pop song domain. Our evaluation shows that this training strategy enables our model, named PiCoGen2, to attain high-quality results, outperforming baselines on both objective and subjective metrics across five pop genres.