RSKNet-MTSP: Effective and Portable Deep Architecture for Speaker Verification

TL;DR

This paper introduces RSKNet-MTSP, a novel CNN architecture for speaker verification that effectively captures long-term and multi-scale speaker features, and a lightweight version suitable for resource-limited applications, demonstrating significant performance improvements.

Contribution

The paper proposes RSKNet-MTSP with residual selective kernel blocks and multi-time-scale pooling, and a lightweight RSKNet-MTSP-L using depthwise separable convolutions and low-rank factorization, advancing speaker verification models.

Findings

RSKNet-MTSP outperforms state-of-the-art methods by 9-26% on public datasets.

RSKNet-MTSP-L achieves comparable results with 17-39% fewer parameters.

Extensive experiments validate the effectiveness of the proposed architecture.

Abstract

The convolutional neural network (CNN) based approaches have shown great success for speaker verification (SV) tasks, where modeling long temporal context and reducing information loss of speaker characteristics are two important challenges significantly affecting the verification performance. Previous works have introduced dilated convolution and multi-scale aggregation methods to address above challenges. However, such methods are also hard to make full use of some valuable information, which make it difficult to substantially improve the verification performance. To address above issues, we construct a novel CNN-based architecture for SV, called RSKNet-MTSP, where a residual selective kernel block (RSKBlock) and a multiple time-scale statistics pooling (MTSP) module are first proposed. The RSKNet-MTSP can capture both long temporal context and neighbouring information, and gather more speaker-discriminative information from multi-scale features. In order to design a portable model for real applications with limited resources, we then present a lightweight version of RSKNet-MTSP, namely RSKNet-MTSP-L, which employs a combination technique associating the depthwise separable convolutions with low-rank factorization of weight matrices. Extensive experiments are conducted on two public SV datasets, VoxCeleb and Speaker in the Wild (SITW). The results demonstrate that 1) RSKNet-MTSP outperforms the state-of-the-art deep embedding architectures by at least 9%-26% in all test sets. 2) RSKNet-MTSP-L achieves competitive performance compared with baseline models with 17%-39% less network parameters. The ablation experiments further illustrate that our proposed approaches can achieve substantial improvement over prior methods.