Target Speaker Voice Activity Detection with Transformers and Its Integration with End-to-End Neural Diarization

TL;DR

This paper introduces a transformer-based target speaker voice activity detection model that handles variable numbers of speakers and improves diarization accuracy, achieving state-of-the-art results on VoxConverse and CALLHOME datasets.

Contribution

The paper proposes a novel transformer architecture for TS-VAD that handles arbitrary speaker counts and integrates it with end-to-end neural diarization, setting new state-of-the-art performance.

Findings

TS-VAD with transformers reduces DER by 11.3% on VoxConverse.

Extended EEND-EDA with transformer TS-VAD reduces DER by 6.9% on CALLHOME.

Achieves new state-of-the-art diarization error rates on both datasets.

Abstract

This paper describes a speaker diarization model based on target speaker voice activity detection (TS-VAD) using transformers. To overcome the original TS-VAD model's drawback of being unable to handle an arbitrary number of speakers, we investigate model architectures that use input tensors with variable-length time and speaker dimensions. Transformer layers are applied to the speaker axis to make the model output insensitive to the order of the speaker profiles provided to the TS-VAD model. Time-wise sequential layers are interspersed between these speaker-wise transformer layers to allow the temporal and cross-speaker correlations of the input speech signal to be captured. We also extend a diarization model based on end-to-end neural diarization with encoder-decoder based attractors (EEND-EDA) by replacing its dot-product-based speaker detection layer with the transformer-based TS-VAD. Experimental results on VoxConverse show that using the transformers for the cross-speaker modeling reduces the diarization error rate (DER) of TS-VAD by 11.3%, achieving a new state-of-the-art (SOTA) DER of 4.57%. Also, our extended EEND-EDA reduces DER by 6.9% on the CALLHOME dataset relative to the original EEND-EDA with a similar model size, achieving a new SOTA DER of 11.18% under a widely used training data setting.