Leveraging the Video-level Semantic Consistency of Event for Audio-visual Event Localization

TL;DR

This paper introduces a novel video-level semantic consistency guidance network for audio-visual event localization, leveraging full-video semantic information to improve accuracy over existing segment-level methods.

Contribution

It proposes an event semantic consistency modeling module with cross-modal and intra-modal components, along with new loss functions, to enhance AVE localization by capturing video-level event semantics.

Findings

Outperforms state-of-the-art methods on AVE dataset

Effective in both fully- and weakly-supervised settings

Improves semantic continuity modeling in AVE localization

Abstract

Audio-visual event (AVE) localization has attracted much attention in recent years. Most existing methods are often limited to independently encoding and classifying each video segment separated from the full video (which can be regarded as the segment-level representations of events). However, they ignore the semantic consistency of the event within the same full video (which can be considered as the video-level representations of events). In contrast to existing methods, we propose a novel video-level semantic consistency guidance network for the AVE localization task. Specifically, we propose an event semantic consistency modeling (ESCM) module to explore video-level semantic information for semantic consistency modeling. It consists of two components: a cross-modal event representation extractor (CERE) and an intra-modal semantic consistency enhancer (ISCE). CERE is proposed to obtain the event semantic information at the video level. Furthermore, ISCE takes video-level event semantics as prior knowledge to guide the model to focus on the semantic continuity of an event within each modality. Moreover, we propose a new negative pair filter loss to encourage the network to filter out the irrelevant segment pairs and a new smooth loss to further increase the gap between different categories of events in the weakly-supervised setting. We perform extensive experiments on the public AVE dataset and outperform the state-of-the-art methods in both fully- and weakly-supervised settings, thus verifying the effectiveness of our method.The code is available at https://github.com/Bravo5542/VSCG.