Metric Learning with Progressive Self-Distillation for Audio-Visual Embedding Learning

TL;DR

This paper introduces a novel audio-visual embedding learning method that combines cross-modal triplet loss with progressive self-distillation to better utilize inherent data distributions and improve alignment beyond label guidance.

Contribution

It proposes a new architecture that integrates progressive self-distillation with triplet loss for enhanced audio-visual embedding learning, capturing complex relationships beyond labels.

Findings

Improved embedding quality demonstrated on benchmark datasets.

Enhanced alignment accuracy between audio and visual modalities.

Outperforms existing label-guided methods in various metrics.

Abstract

Metric learning projects samples into an embedded space, where similarities and dissimilarities are quantified based on their learned representations. However, existing methods often rely on label-guided representation learning, where representations of different modalities, such as audio and visual data, are aligned based on annotated labels. This approach tends to underutilize latent complex features and potential relationships inherent in the distributions of audio and visual data that are not directly tied to the labels, resulting in suboptimal performance in audio-visual embedding learning. To address this issue, we propose a novel architecture that integrates cross-modal triplet loss with progressive self-distillation. Our method enhances representation learning by leveraging inherent distributions and dynamically refining soft audio-visual alignments -- probabilistic alignments between audio and visual data that capture the inherent relationships beyond explicit labels. Specifically, the model distills audio-visual distribution-based knowledge from annotated labels in a subset of each batch. This self-distilled knowledge is used t