Mutual Learning for Acoustic Matching and Dereverberation via Visual Scene-driven Diffusion

TL;DR

This paper introduces MVSD, a mutual learning diffusion-based framework that jointly improves visual acoustic matching and dereverberation by leveraging their reciprocal relationship and unpaired data, enhancing audio-visual scene consistency.

Contribution

The paper proposes a novel mutual learning framework using diffusion models for joint acoustic matching and dereverberation, overcoming data scarcity and training issues of prior methods.

Findings

Improves performance of reverberator and dereverberator

Better matching of visual scenarios in audio tasks

Effective with unpaired data

Abstract

Visual acoustic matching (VAM) is pivotal for enhancing the immersive experience, and the task of dereverberation is effective in improving audio intelligibility. Existing methods treat each task independently, overlooking the inherent reciprocity between them. Moreover, these methods depend on paired training data, which is challenging to acquire, impeding the utilization of extensive unpaired data. In this paper, we introduce MVSD, a mutual learning framework based on diffusion models. MVSD considers the two tasks symmetrically, exploiting the reciprocal relationship to facilitate learning from inverse tasks and overcome data scarcity. Furthermore, we employ the diffusion model as foundational conditional converters to circumvent the training instability and over-smoothing drawbacks of conventional GAN architectures. Specifically, MVSD employs two converters: one for VAM called reverberator and one for dereverberation called dereverberator. The dereverberator judges whether the reverberation audio generated by reverberator sounds like being in the conditional visual scenario, and vice versa. By forming a closed loop, these two converters can generate informative feedback signals to optimize the inverse tasks, even with easily acquired one-way unpaired data. Extensive experiments on two standard benchmarks, i.e., SoundSpaces-Speech and Acoustic AVSpeech, exhibit that our framework can improve the performance of the reverberator and dereverberator and better match specified visual scenarios.