Noise-Robust Target-Speaker Voice Activity Detection Through Self-Supervised Pretraining

TL;DR

This paper introduces a self-supervised pretraining method called DN-APC to improve target-speaker voice activity detection in noisy environments, demonstrating enhanced robustness and performance over traditional models.

Contribution

The paper proposes a novel SSL pretraining framework for TS-VAD that reduces reliance on labeled data and improves noise robustness, with analysis of different speaker conditioning methods.

Findings

DN-APC improves TS-VAD performance by approximately 2% in noisy conditions.

FiLM conditioning yields the best overall performance among tested methods.

Representation analysis shows robust initial speech/non-speech features from pretraining.

Abstract

Target-Speaker Voice Activity Detection (TS-VAD) is the task of detecting the presence of speech from a known target-speaker in an audio frame. Recently, deep neural network-based models have shown good performance in this task. However, training these models requires extensive labelled data, which is costly and time-consuming to obtain, particularly if generalization to unseen environments is crucial. To mitigate this, we propose a causal, Self-Supervised Learning (SSL) pretraining framework, called Denoising Autoregressive Predictive Coding (DN-APC), to enhance TS-VAD performance in noisy conditions. We also explore various speaker conditioning methods and evaluate their performance under different noisy conditions. Our experiments show that DN-APC improves performance in noisy conditions, with a general improvement of approx. 2% in both seen and unseen noise. Additionally, we find that FiLM conditioning provides the best overall performance. Representation analysis via tSNE plots reveals robust initial representations of speech and non-speech from pretraining. This underscores the effectiveness of SSL pretraining in improving the robustness and performance of TS-VAD models in noisy environments.