Text-Driven Foley Sound Generation With Latent Diffusion Model

TL;DR

This paper introduces a text-driven diffusion model for Foley sound generation that leverages transfer learning, improved text embeddings, and candidate selection to produce high-quality background sounds from text descriptions.

Contribution

It presents a novel diffusion-based system with a trainable text embedding layer and candidate audio selection, advancing text-to-sound generation methods.

Findings

Achieved 1st place in DCASE Challenge 2023 Task 7

Significant performance improvements from proposed techniques

Effective use of transfer learning and candidate selection

Abstract

Foley sound generation aims to synthesise the background sound for multimedia content. Previous models usually employ a large development set with labels as input (e.g., single numbers or one-hot vector). In this work, we propose a diffusion model based system for Foley sound generation with text conditions. To alleviate the data scarcity issue, our model is initially pre-trained with large-scale datasets and fine-tuned to this task via transfer learning using the contrastive language-audio pertaining (CLAP) technique. We have observed that the feature embedding extracted by the text encoder can significantly affect the performance of the generation model. Hence, we introduce a trainable layer after the encoder to improve the text embedding produced by the encoder. In addition, we further refine the generated waveform by generating multiple candidate audio clips simultaneously and selecting the best one, which is determined in terms of the similarity score between the embedding of the candidate clips and the embedding of the target text label. Using the proposed method, our system ranks ${1}^{st}$ among the systems submitted to DCASE Challenge 2023 Task 7. The results of the ablation studies illustrate that the proposed techniques significantly improve sound generation performance. The codes for implementing the proposed system are available online.