DiffRhythm+: Controllable and Flexible Full-Length Song Generation with Preference Optimization

TL;DR

DiffRhythm+ is a diffusion-based model that generates full-length, expressive songs with enhanced controllability and diversity by using a balanced dataset, multi-modal style conditioning, and preference optimization.

Contribution

It introduces DiffRhythm+, which improves controllability, diversity, and quality in full-length song generation through dataset balancing, multi-modal style conditioning, and user preference-guided optimization.

Findings

Significant improvements in naturalness and musical expressiveness.

Enhanced controllability over musical styles via multi-modal conditioning.

Higher listener satisfaction and arrangement complexity.

Abstract

Songs, as a central form of musical art, exemplify the richness of human intelligence and creativity. While recent advances in generative modeling have enabled notable progress in long-form song generation, current systems for full-length song synthesis still face major challenges, including data imbalance, insufficient controllability, and inconsistent musical quality. DiffRhythm, a pioneering diffusion-based model, advanced the field by generating full-length songs with expressive vocals and accompaniment. However, its performance was constrained by an unbalanced model training dataset and limited controllability over musical style, resulting in noticeable quality disparities and restricted creative flexibility. To address these limitations, we propose DiffRhythm+, an enhanced diffusion-based framework for controllable and flexible full-length song generation. DiffRhythm+ leverages a substantially expanded and balanced training dataset to mitigate issues such as repetition and omission of lyrics, while also fostering the emergence of richer musical skills and expressiveness. The framework introduces a multi-modal style conditioning strategy, enabling users to precisely specify musical styles through both descriptive text and reference audio, thereby significantly enhancing creative control and diversity. We further introduce direct performance optimization aligned with user preferences, guiding the model toward consistently preferred outputs across evaluation metrics. Extensive experiments demonstrate that DiffRhythm+ achieves significant improvements in naturalness, arrangement complexity, and listener satisfaction over previous systems.