Speak in the Scene: Diffusion-based Acoustic Scene Transfer toward Immersive Speech Generation

TL;DR

This paper presents Acoustic Scene Transfer (AST), a new generative speech task that modifies the acoustic environment of speech signals to create immersive listening experiences, using a diffusion model conditioned by scene embeddings.

Contribution

It introduces the AST task and proposes AST-LDM, a diffusion-based model that transfers acoustic scenes while preserving speech content, validated through comprehensive experiments.

Findings

AST-LDM effectively transfers acoustic scenes with high fidelity.

Objective and subjective tests confirm the model's feasibility and quality.

The approach enhances immersive speech perception in diverse environments.

Abstract

This paper introduces a novel task in generative speech processing, Acoustic Scene Transfer (AST), which aims to transfer acoustic scenes of speech signals to diverse environments. AST promises an immersive experience in speech perception by adapting the acoustic scene behind speech signals to desired environments. We propose AST-LDM for the AST task, which generates speech signals accompanied by the target acoustic scene of the reference prompt. Specifically, AST-LDM is a latent diffusion model conditioned by CLAP embeddings that describe target acoustic scenes in either audio or text modalities. The contributions of this paper include introducing the AST task and implementing its baseline model. For AST-LDM, we emphasize its core framework, which is to preserve the input speech and generate audio consistently with both the given speech and the target acoustic environment. Experiments, including objective and subjective tests, validate the feasibility and efficacy of our approach.