Latent Diffusion Model Based Foley Sound Generation System For DCASE Challenge 2023 Task 7

TL;DR

This paper introduces a diffusion-based Foley sound generation system for the DCASE 2023 challenge, leveraging transfer learning, language-audio embeddings, and filtering strategies to improve sound synthesis quality.

Contribution

The system combines AudioLDM with CLAP embeddings and filtering to enhance Foley sound generation, achieving significant performance improvements over baseline methods.

Findings

Achieved an average FAD score of 4.765, outperforming the baseline of 9.7.

Demonstrated the effectiveness of CLAP embeddings in improving sound quality.

Utilized transfer learning to adapt a large-scale pre-trained model to the Foley sound synthesis task.

Abstract

Foley sound presents the background sound for multimedia content and the generation of Foley sound involves computationally modelling sound effects with specialized techniques. In this work, we proposed a system for DCASE 2023 challenge task 7: Foley Sound Synthesis. The proposed system is based on AudioLDM, which is a diffusion-based text-to-audio generation model. To alleviate the data-hungry problem, the system first trained with large-scale datasets and then downstreamed into this DCASE task via transfer learning. Through experiments, we found out that the feature extracted by the encoder can significantly affect the performance of the generation model. Hence, we improve the results by leveraging the input label with related text embedding features obtained by a significant language model, i.e., contrastive language-audio pertaining (CLAP). In addition, we utilize a filtering strategy to further refine the output, i.e. by selecting the best results from the candidate clips generated in terms of the similarity score between the sound and target labels. The overall system achieves a Frechet audio distance (FAD) score of 4.765 on average among all seven different classes, substantially outperforming the baseline system which performs a FAD score of 9.7.