Audio-Visual Speech Separation via Bottleneck Iterative Network

TL;DR

This paper introduces the Bottleneck Iterative Network (BIN), a lightweight, iterative fusion approach for audio-visual speech separation that enhances model capacity and outperforms benchmarks while reducing training and inference time.

Contribution

The paper proposes BIN, a novel iterative fusion method using fusion tokens, improving speech separation performance without increasing model size significantly.

Findings

BIN outperforms state-of-the-art models on NTCD-TIMIT and LRS3+WHAM! datasets.

BIN reduces training and inference time by over 50%.

BIN effectively balances model capacity and computational efficiency.

Abstract

Integration of information from non-auditory cues can significantly improve the performance of speech-separation models. Often such models use deep modality-specific networks to obtain unimodal features, and risk being too costly or lightweight but lacking capacity. In this work, we present an iterative representation refinement approach called Bottleneck Iterative Network (BIN), a technique that repeatedly progresses through a lightweight fusion block, while bottlenecking fusion representations by fusion tokens. This helps improve the capacity of the model, while avoiding major increase in model size and balancing between the model performance and training cost. We test BIN on challenging noisy audio-visual speech separation tasks, and show that our approach consistently outperforms state-of-the-art benchmark models with respect to SI-SDRi on NTCD-TIMIT and LRS3+WHAM! datasets, while simultaneously achieving a reduction of more than 50% in training and GPU inference time across nearly all settings.