Rethinking Leveraging Pre-Trained Multi-Layer Representations for Speaker Verification

TL;DR

This paper introduces Layer Attentive Pooling (LAP), a dynamic and effective method for aggregating multi-layer features from pre-trained speech models, significantly improving speaker verification performance.

Contribution

The paper proposes LAP, a novel dynamic layer aggregation technique, and a lightweight backend model combining LAP and ASTP, achieving state-of-the-art results with reduced training time.

Findings

LAP outperforms static averaging methods in speaker verification.

The proposed architecture achieves state-of-the-art performance on VoxCeleb.

Dynamic weighting captures speaker characteristics more effectively.

Abstract

Recent speaker verification studies have achieved notable success by leveraging layer-wise output from pre-trained Transformer models. However, few have explored the advancements in aggregating these multi-level features beyond the static weighted average. We present Layer Attentive Pooling (LAP), a novel strategy for aggregating inter-layer representations from pre-trained speech models for speaker verification. LAP assesses the significance of each layer from multiple perspectives time-dynamically, and employs max pooling instead of averaging. Additionally, we propose a lightweight backend speaker model comprising LAP and Attentive Statistical Temporal Pooling (ASTP) to extract speaker embeddings from pre-trained model output. Experiments on the VoxCeleb benchmark reveal that our compact architecture achieves state-of-the-art performance while greatly reducing the training time. We further analyzed LAP design and its dynamic weighting mechanism for capturing speaker characteristics.