Multi-query multi-head attention pooling and Inter-topK penalty for speaker verification

TL;DR

This paper introduces the MQMHA pooling and inter-topK penalty methods for speaker verification, combining diversified attention mechanisms and inter-class discriminability enhancements to achieve state-of-the-art results.

Contribution

The paper proposes a novel multi-query multi-head attention pooling and an inter-topK penalty to improve speaker verification performance.

Findings

Achieved state-of-the-art results on VoxCeleb test sets.

Demonstrated improved inter-class discriminability.

Enhanced speaker representation diversity.

Abstract

This paper describes the multi-query multi-head attention (MQMHA) pooling and inter-topK penalty methods which were first proposed in our submitted system description for VoxCeleb speaker recognition challenge (VoxSRC) 2021. Most multi-head attention pooling mechanisms either attend to the whole feature through multiple heads or attend to several split parts of the whole feature. Our proposed MQMHA combines both these two mechanisms and gain more diversified information. The margin-based softmax loss functions are commonly adopted to obtain discriminative speaker representations. To further enhance the inter-class discriminability, we propose a method that adds an extra inter-topK penalty on some confused speakers. By adopting both the MQMHA and inter-topK penalty, we achieved state-of-the-art performance in all of the public VoxCeleb test sets.