Cross-Modal Attention Consistency for Video-Audio Unsupervised Learning

TL;DR

This paper introduces Cross-Modal Attention Consistency (CMAC), a novel unsupervised learning method that aligns visual and audio regional attention to improve video-audio representation learning.

Contribution

It proposes a new pretext task, CMAC, that enforces bidirectional local correspondence between visual and audio signals within a contrastive learning framework.

Findings

Improves state-of-the-art performance on multiple benchmarks.

Effectively aligns visual and audio regional attention.

Enhances unsupervised video-audio representation learning.

Abstract

Cross-modal correlation provides an inherent supervision for video unsupervised representation learning. Existing methods focus on distinguishing different video clips by visual and audio representations. We human visual perception could attend to regions where sounds are made, and our auditory perception could also ground their frequencies of sounding objects, which we call bidirectional local correspondence. Such supervision is intuitive but not well explored in the contrastive learning framework. This paper introduces a pretext task, Cross-Modal Attention Consistency (CMAC), for exploring the bidirectional local correspondence property. The CMAC approach aims to align the regional attention generated purely from the visual signal with the target attention generated under the guidance of acoustic signal, and do a similar alignment for frequency grounding on the acoustic attention. Accompanied by a remoulded cross-modal contrastive loss where we consider additional within-modal interactions, the CMAC approach works effectively for enforcing the bidirectional alignment. Extensive experiments on six downstream benchmarks demonstrate that CMAC can improve the state-of-the-art performance on both visual and audio modalities.