Language-Guided Contrastive Audio-Visual Masked Autoencoder with Automatically Generated Audio-Visual-Text Triplets from Videos

TL;DR

This paper introduces LG-CAV-MAE, a novel contrastive autoencoder that leverages language guidance and automatically generated audio-visual-text triplets to enhance multi-modal representation learning from unlabeled videos.

Contribution

It proposes a new framework integrating a pretrained text encoder with contrastive masked autoencoders and an automatic triplet generation method for improved audio-visual learning.

Findings

Achieves up to 5.6% improvement in retrieval recall@10.

Improves classification accuracy by 3.2%.

Outperforms existing methods significantly.

Abstract

In this paper, we propose Language-Guided Contrastive Audio-Visual Masked Autoencoders (LG-CAV-MAE) to improve audio-visual representation learning. LG-CAV-MAE integrates a pretrained text encoder into contrastive audio-visual masked autoencoders, enabling the model to learn across audio, visual and text modalities. To train LG-CAV-MAE, we introduce an automatic method to generate audio-visual-text triplets from unlabeled videos. We first generate frame-level captions using an image captioning model and then apply CLAP-based filtering to ensure strong alignment between audio and captions. This approach yields high-quality audio-visual-text triplets without requiring manual annotations. We evaluate LG-CAV-MAE on audio-visual retrieval tasks, as well as an audio-visual classification task. Our method significantly outperforms existing approaches, achieving up to a 5.6% improvement in recall@10 for retrieval tasks and a 3.2% improvement for the classification task.