Improving Transformer-based Networks With Locality For Automatic Speaker Verification

TL;DR

This paper introduces two novel Transformer-based models with enhanced locality mechanisms for speaker verification, significantly improving accuracy over existing Transformer and CNN models on multiple datasets.

Contribution

It proposes the Locality-Enhanced Conformer and Speaker Swin Transformer, integrating locality modeling into Transformer architectures for improved speaker embedding extraction.

Findings

Achieved 0.75% EER on VoxCeleb 1, outperforming prior models.

Reduced EER by 14.6% relative on MS-internal dataset compared to Res2Net50.

Demonstrated effectiveness of locality-enhanced Transformers in speaker verification.

Abstract

Recently, Transformer-based architectures have been explored for speaker embedding extraction. Although the Transformer employs the self-attention mechanism to efficiently model the global interaction between token embeddings, it is inadequate for capturing short-range local context, which is essential for the accurate extraction of speaker information. In this study, we enhance the Transformer with the enhanced locality modeling in two directions. First, we propose the Locality-Enhanced Conformer (LE-Confomer) by introducing depth-wise convolution and channel-wise attention into the Conformer blocks. Second, we present the Speaker Swin Transformer (SST) by adapting the Swin Transformer, originally proposed for vision tasks, into speaker embedding network. We evaluate the proposed approaches on the VoxCeleb datasets and a large-scale Microsoft internal multilingual (MS-internal) dataset. The proposed models achieve 0.75% EER on VoxCeleb 1 test set, outperforming the previously proposed Transformer-based models and CNN-based models, such as ResNet34 and ECAPA-TDNN. When trained on the MS-internal dataset, the proposed models achieve promising results with 14.6% relative reduction in EER over the Res2Net50 model.