Duality Temporal-channel-frequency Attention Enhanced Speaker Representation Learning

TL;DR

This paper introduces a novel Duality Temporal-Channel-Frequency (DTCF) attention mechanism that enhances speaker representation learning by capturing mutual interactions across temporal, channel, and frequency scales, improving verification performance.

Contribution

The paper proposes the DTCF attention module that models interactions among temporal, channel, and frequency features, leading to more discriminative speaker representations in CNN-based networks.

Findings

DTCF attention reduces EER and minDCF on CN-Celeb and VoxCeleb datasets.

ResNet34-DTCF outperforms ResNet34-SE in speaker verification tasks.

Significant improvements in speaker verification metrics demonstrate effectiveness.

Abstract

The use of channel-wise attention in CNN based speaker representation networks has achieved remarkable performance in speaker verification (SV). But these approaches do simple averaging on time and frequency feature maps before channel-wise attention learning and ignore the essential mutual interaction among temporal, channel as well as frequency scales. To address this problem, we propose the Duality Temporal-Channel-Frequency (DTCF) attention to re-calibrate the channel-wise features with aggregation of global context on temporal and frequency dimensions. Specifically, the duality attention - time-channel (T-C) attention as well as frequency-channel (F-C) attention - aims to focus on salient regions along the T-C and F-C feature maps that may have more considerable impact on the global context, leading to more discriminative speaker representations. We evaluate the effectiveness of the proposed DTCF attention on the CN-Celeb and VoxCeleb datasets. On the CN-Celeb evaluation set, the EER/minDCF of ResNet34-DTCF are reduced by 0.63%/0.0718 compared with those of ResNet34-SE. On VoxCeleb1-O, VoxCeleb1-E and VoxCeleb1-H evaluation sets, the EER/minDCF of ResNet34-DTCF achieve 0.36%/0.0263, 0.39%/0.0382 and 0.74%/0.0753 reductions compared with those of ResNet34-SE.