Training Strategies for Modality Dropout Resilient Multi-Modal Target Speaker Extraction

TL;DR

This paper introduces a modality dropout training strategy for multi-modal target speaker extraction, enhancing robustness against modality dominance and improving performance when one modality is unavailable.

Contribution

The study proposes modality dropout training as a novel approach to improve robustness in multi-modal speaker extraction systems, outperforming standard and multi-task training methods.

Findings

MDT outperforms standard and MTT strategies across experiments.

Models with MDT are less affected by normalization layer choices.

MDT-trained systems are robust to using speech as enrollment signals.

Abstract

The primary goal of multi-modal TSE (MTSE) is to extract a target speaker from a speech mixture using complementary information from different modalities, such as audio enrolment and visual feeds corresponding to the target speaker. MTSE systems are expected to perform well even when one of the modalities is unavailable. In practice, the systems often suffer from modality dominance, where one of the modalities outweighs the others, thereby limiting robustness. Our study investigates training strategies and the effect of architectural choices, particularly the normalization layers, in yielding a robust MTSE system in both non-causal and causal configurations. In particular, we propose the use of modality dropout training (MDT) as a superior strategy to standard and multi-task training (MTT) strategies. Experiments conducted on two-speaker mixtures from the LRS3 dataset show the MDT strategy to be effective irrespective of the employed normalization layer. In contrast, the models trained with the standard and MTT strategies are susceptible to modality dominance, and their performance depends on the chosen normalization layer. Additionally, we demonstrate that the system trained with MDT strategy is robust to using extracted speech as the enrollment signal, highlighting its potential applicability in scenarios where the target speaker is not enrolled.