Resonate: Reinforcing Text-to-Audio Generation via Online Feedback from Large Audio Language Models

TL;DR

This paper introduces Resonate, a novel online reinforcement learning approach for text-to-audio generation that leverages large audio language models, significantly improving quality and alignment over previous offline methods.

Contribution

It adapts online Group Relative Policy Optimization for TTA models and demonstrates superior performance using rewards from large audio language models.

Findings

Resonate outperforms offline RL methods in TTA quality.

Resonate achieves state-of-the-art results on TTA-Bench.

The model uses only 470M parameters.

Abstract

Reinforcement Learning (RL) has become an effective paradigm for enhancing Large Language Models (LLMs) and visual generative models. However, its application in text-to-audio (TTA) generation remains largely under-explored. Prior work typically employs offline methods like Direct Preference Optimization (DPO) and leverages Contrastive Language-Audio Pretraining (CLAP) models as reward functions. In this study, we investigate the integration of online Group Relative Policy Optimization (GRPO) into TTA generation. We adapt the algorithm for Flow Matching-based audio models and demonstrate that online RL significantly outperforms its offline counterparts. Furthermore, we incorporate rewards derived from Large Audio Language Models (LALMs), which can provide fine-grained scoring signals that are better aligned with human perception. With only 470M parameters, our final model, \textbf{Resonate}, establishes a new SOTA on TTA-Bench in terms of both audio quality and semantic alignment.