SongEditor: Adapting Zero-Shot Song Generation Language Model as a Multi-Task Editor

TL;DR

SongEditor introduces a novel multi-task song editing framework that leverages language models and diffusion techniques to enable flexible modifications and synthesis of songs, including lyrics, vocals, and accompaniment.

Contribution

It is the first to incorporate editing capabilities into language-model-based song generation, allowing segment-wise and track-wise modifications for more versatile song production.

Findings

Achieves high performance in end-to-end song editing tasks

Demonstrates effectiveness through objective and subjective metrics

Supports flexible editing of lyrics, vocals, and accompaniment

Abstract

The emergence of novel generative modeling paradigms, particularly audio language models, has significantly advanced the field of song generation. Although state-of-the-art models are capable of synthesizing both vocals and accompaniment tracks up to several minutes long concurrently, research about partial adjustments or editing of existing songs is still underexplored, which allows for more flexible and effective production. In this paper, we present SongEditor, the first song editing paradigm that introduces the editing capabilities into language-modeling song generation approaches, facilitating both segment-wise and track-wise modifications. SongEditor offers the flexibility to adjust lyrics, vocals, and accompaniments, as well as synthesizing songs from scratch. The core components of SongEditor include a music tokenizer, an autoregressive language model, and a diffusion generator, enabling generating an entire section, masked lyrics, or even separated vocals and background music. Extensive experiments demonstrate that the proposed SongEditor achieves exceptional performance in end-to-end song editing, as evidenced by both objective and subjective metrics. Audio samples are available in https://cypress-yang.github.io/SongEditor_demo/.