CACE-Net: Co-guidance Attention and Contrastive Enhancement for Effective Audio-Visual Event Localization

TL;DR

CACE-Net introduces a bi-directional co-guidance attention mechanism and contrastive enhancement to improve audio-visual event localization accuracy in complex videos.

Contribution

The paper presents a novel co-guidance attention mechanism and contrastive enhancement techniques for better multimodal feature integration and event discrimination.

Findings

Sets new benchmark on AVE dataset

Improves event classification accuracy

Enhances subtle difference detection between event and background

Abstract

The audio-visual event localization task requires identifying concurrent visual and auditory events from unconstrained videos within a network model, locating them, and classifying their category. The efficient extraction and integration of audio and visual modal information have always been challenging in this field. In this paper, we introduce CACE-Net, which differs from most existing methods that solely use audio signals to guide visual information. We propose an audio-visual co-guidance attention mechanism that allows for adaptive bi-directional cross-modal attentional guidance between audio and visual information, thus reducing inconsistencies between modalities. Moreover, we have observed that existing methods have difficulty distinguishing between similar background and event and lack the fine-grained features for event classification. Consequently, we employ background-event contrast enhancement to increase the discrimination of fused feature and fine-tuned pre-trained model to extract more refined and discernible features from complex multimodal inputs. Specifically, we have enhanced the model's ability to discern subtle differences between event and background and improved the accuracy of event classification in our model. Experiments on the AVE dataset demonstrate that CACE-Net sets a new benchmark in the audio-visual event localization task, proving the effectiveness of our proposed methods in handling complex multimodal learning and event localization in unconstrained videos. Code is available at https://github.com/Brain-Cog-Lab/CACE-Net.