Mel-Refine: A Plug-and-Play Approach to Refine Mel-Spectrogram in Audio Generation

TL;DR

Mel-Refine is a plug-and-play method that improves the detail and texture of Mel-spectrograms in text-to-audio models without retraining, significantly enhancing audio quality.

Contribution

The paper introduces Mel-Refine, a novel inference-time technique that refines Mel-spectrograms in diffusion-based TTA models, requiring no additional training.

Findings

Boosts Tango2 model performance by 25%

Enhances Mel-spectrogram detail and texture

Compatible with any diffusion-based TTA architecture

Abstract

Text-to-audio (TTA) model is capable of generating diverse audio from textual prompts. However, most mainstream TTA models, which predominantly rely on Mel-spectrograms, still face challenges in producing audio with rich content. The intricate details and texture required in Mel-spectrograms for such audio often surpass the models' capacity, leading to outputs that are blurred or lack coherence. In this paper, we begin by investigating the critical role of U-Net in Mel-spectrogram generation. Our analysis shows that in U-Net structure, high-frequency components in skip-connections and the backbone influence texture and detail, while low-frequency components in the backbone are critical for the diffusion denoising process. We further propose ``Mel-Refine'', a plug-and-play approach that enhances Mel-spectrogram texture and detail by adjusting different component weights during inference. Our method requires no additional training or fine-tuning and is fully compatible with any diffusion-based TTA architecture. Experimental results show that our approach boosts performance metrics of the latest TTA model Tango2 by 25\%, demonstrating its effectiveness.