AdVerb: Visually Guided Audio Dereverberation

TL;DR

AdVerb is a novel audio-visual dereverberation framework that leverages visual cues and a geometry-aware transformer to improve the estimation of clean audio from reverberant recordings, outperforming existing methods.

Contribution

It introduces a geometry-aware cross-modal transformer architecture that effectively combines visual and audio data for dereverberation, a novel approach in this domain.

Findings

Significantly outperforms traditional audio-only methods.

Achieves 18%-82% improvements on speech tasks.

Attains low RT60 error scores on AVSpeech dataset.

Abstract

We present AdVerb, a novel audio-visual dereverberation framework that uses visual cues in addition to the reverberant sound to estimate clean audio. Although audio-only dereverberation is a well-studied problem, our approach incorporates the complementary visual modality to perform audio dereverberation. Given an image of the environment where the reverberated sound signal has been recorded, AdVerb employs a novel geometry-aware cross-modal transformer architecture that captures scene geometry and audio-visual cross-modal relationship to generate a complex ideal ratio mask, which, when applied to the reverberant audio predicts the clean sound. The effectiveness of our method is demonstrated through extensive quantitative and qualitative evaluations. Our approach significantly outperforms traditional audio-only and audio-visual baselines on three downstream tasks: speech enhancement, speech recognition, and speaker verification, with relative improvements in the range of 18% - 82% on the LibriSpeech test-clean set. We also achieve highly satisfactory RT60 error scores on the AVSpeech dataset.