SSAVSV: Towards Unified Model for Self-Supervised Audio-Visual Speaker Verification

TL;DR

This paper introduces a self-supervised audiovisual speaker verification framework using a shared vision transformer backbone, enabling efficient, modality-agnostic verification without labeled data, and demonstrating competitive performance.

Contribution

It presents a unified self-supervised learning approach with a single backbone for audio and visual inputs, reducing computational costs and handling missing modalities.

Findings

Achieves competitive verification performance without labeled data.

Reduces computational costs compared to traditional methods.

Handles missing modalities effectively.

Abstract

Conventional audio-visual methods for speaker verification rely on large amounts of labeled data and separate modality-specific architectures, which is computationally expensive, limiting their scalability. To address these problems, we propose a self-supervised learning framework based on contrastive learning with asymmetric masking and masked data modeling to obtain robust audiovisual feature representations. In particular, we employ a unified framework for self-supervised audiovisual speaker verification using a single shared backbone for audio and visual inputs, leveraging the versatility of vision transformers. The proposed unified framework can handle audio, visual, or audiovisual inputs using a single shared vision transformer backbone during training and testing while being computationally efficient and robust to missing modalities. Extensive experiments demonstrate that our method achieves competitive performance without labeled data while reducing computational costs compared to traditional approaches.