AudioRAG+: Feedback-driven Retrieval-augmented Audio Generation with Large Audio Language Models

TL;DR

AudioRAG+ introduces a feedback-driven retrieval-augmented approach using Large Audio Language Models to improve text-to-audio generation by identifying and incorporating missing sound events, outperforming existing methods.

Contribution

It presents a novel feedback-driven RAG method leveraging LALMs for more accurate sound event synthesis in TTA tasks, avoiding training from scratch.

Findings

Enhanced sound event detection in generated audio.

Outperforms existing RAG-based TTA methods.

Improves generalization across different models.

Abstract

We propose a general feedback-driven retrieval-augmented generation (RAG) approach that leverages Large Audio Language Models (LALMs) to address the missing or imperfect synthesis of specific sound events in text-to-audio (TTA) generation. Unlike previous RAG-based TTA methods that typically train specialized models from scratch, we utilize LALMs to analyze audio generation outputs, retrieve concepts that pre-trained models struggle to generate from an external database, and incorporate the retrieved information into the generation process. Experimental results show that our method not only enhances the ability of LALMs to identify missing sound events but also delivers improvements across different models, outperforming existing RAG-specialized approaches.