DiffRhythm 2: Efficient and High Fidelity Song Generation via Block Flow Matching

TL;DR

DiffRhythm 2 presents an end-to-end, high-fidelity song generation framework that aligns lyrics with vocals, maintains long-term coherence, and effectively incorporates diverse human preferences through novel optimization techniques.

Contribution

The paper introduces DiffRhythm 2, a semi-autoregressive model with block flow matching and a music VAE, enabling faithful lyric alignment, efficient long-sequence generation, and robust multi-preference optimization.

Findings

Achieves high-quality, long-duration song generation with accurate lyric alignment.

Reduces performance degradation in multi-preference optimization via cross-pair preference method.

Maintains high audio fidelity with low frame rate music VAE.

Abstract

Generating full-length, high-quality songs is challenging, as it requires maintaining long-term coherence both across text and music modalities and within the music modality itself. Existing non-autoregressive (NAR) frameworks, while capable of producing high-quality songs, often struggle with the alignment between lyrics and vocal. Concurrently, catering to diverse musical preferences necessitates reinforcement learning from human feedback (RLHF). However, existing methods often rely on merging multiple models during multi-preference optimization, which results in significant performance degradation. To address these challenges, we introduce DiffRhythm 2, an end-to-end framework designed for high-fidelity, controllable song generation. To tackle the lyric alignment problem, DiffRhythm 2 employs a semi-autoregressive architecture based on block flow matching. This design enables faithful alignment of lyrics to singing vocals without relying on external labels and constraints, all while preserving the high generation quality and efficiency of NAR models. To make this framework computationally tractable for long sequences, we implement a music variational autoencoder (VAE) that achieves a low frame rate of 5 Hz while still enabling high-fidelity audio reconstruction. In addition, to overcome the limitations of multi-preference optimization in RLHF, we propose cross-pair preference optimization. This method effectively mitigates the performance drop typically associated with model merging, allowing for more robust optimization across diverse human preferences. We further enhance musicality and structural coherence by introducing stochastic block representation alignment loss.