Learning Audio-Visual Correlations from Variational Cross-Modal Generation

TL;DR

This paper introduces a self-supervised variational autoencoder framework that learns audio-visual correlations from cross-modal generation, enabling effective downstream tasks like localization and retrieval without labeled data.

Contribution

It proposes a novel MS-VAE model with Wasserstein constraint to learn intrinsic audio-visual correlations in a self-supervised manner.

Findings

Effective learning of audio-visual correlations without labels.

Competitive performance on downstream tasks.

Versatile application in localization and retrieval.

Abstract

People can easily imagine the potential sound while seeing an event. This natural synchronization between audio and visual signals reveals their intrinsic correlations. To this end, we propose to learn the audio-visual correlations from the perspective of cross-modal generation in a self-supervised manner, the learned correlations can be then readily applied in multiple downstream tasks such as the audio-visual cross-modal localization and retrieval. We introduce a novel Variational AutoEncoder (VAE) framework that consists of Multiple encoders and a Shared decoder (MS-VAE) with an additional Wasserstein distance constraint to tackle the problem. Extensive experiments demonstrate that the optimized latent representation of the proposed MS-VAE can effectively learn the audio-visual correlations and can be readily applied in multiple audio-visual downstream tasks to achieve competitive performance even without any given label information during training.