Audio-Visual Speaker Verification via Joint Cross-Attention

TL;DR

This paper introduces a novel cross-attention based method for audio-visual speaker verification, leveraging inter- and intra-modal relationships to improve accuracy over existing fusion techniques.

Contribution

It proposes a cross-modal joint attention mechanism that fully exploits inter- and intra-modal information for enhanced speaker verification performance.

Findings

Significantly outperforms state-of-the-art methods on Voxceleb1.

Effectively captures inter- and intra-modal relationships.

Improves accuracy of audio-visual speaker verification.

Abstract

Speaker verification has been widely explored using speech signals, which has shown significant improvement using deep models. Recently, there has been a surge in exploring faces and voices as they can offer more complementary and comprehensive information than relying only on a single modality of speech signals. Though current methods in the literature on the fusion of faces and voices have shown improvement over that of individual face or voice modalities, the potential of audio-visual fusion is not fully explored for speaker verification. Most of the existing methods based on audio-visual fusion either rely on score-level fusion or simple feature concatenation. In this work, we have explored cross-modal joint attention to fully leverage the inter-modal complementary information and the intra-modal information for speaker verification. Specifically, we estimate the cross-attention weights based on the correlation between the joint feature presentation and that of the individual feature representations in order to effectively capture both intra-modal as well inter-modal relationships among the faces and voices. We have shown that efficiently leveraging the intra- and inter-modal relationships significantly improves the performance of audio-visual fusion for speaker verification. The performance of the proposed approach has been evaluated on the Voxceleb1 dataset. Results show that the proposed approach can significantly outperform the state-of-the-art methods of audio-visual fusion for speaker verification.