ESG-Net: Event-Aware Semantic Guided Network for Dense Audio-Visual Event Localization

TL;DR

ESG-Net introduces a multi-stage semantic guidance and multi-event dependency modeling approach to improve dense audio-visual event localization, achieving superior accuracy with fewer parameters.

Contribution

The paper proposes ESG-Net, which incorporates hierarchical semantic understanding and adaptive event dependency extraction, addressing semantic gaps and event correlation challenges in DAVE.

Findings

Outperforms state-of-the-art methods on benchmark datasets.

Reduces model parameters and computational load significantly.

Enhances hierarchical semantic understanding of audio-visual events.

Abstract

Dense audio-visual event localization (DAVE) aims to identify event categories and locate the temporal boundaries in untrimmed videos. Most studies only employ event-related semantic constraints on the final outputs, lacking cross-modal semantic bridging in intermediate layers. This causes modality semantic gap for further fusion, making it difficult to distinguish between event-related content and irrelevant background content. Moreover, they rarely consider the correlations between events, which limits the model to infer concurrent events among complex scenarios. In this paper, we incorporate multi-stage semantic guidance and multi-event relationship modeling, which respectively enable hierarchical semantic understanding of audio-visual events and adaptive extraction of event dependencies, thereby better focusing on event-related information. Specifically, our eventaware semantic guided network (ESG-Net) includes a early semantics interaction (ESI) module and a mixture of dependency experts (MoDE) module. ESI applys multi-stage semantic guidance to explicitly constrain the model in learning semantic information through multi-modal early fusion and several classification loss functions, ensuring hierarchical understanding of event-related content. MoDE promotes the extraction of multi-event dependencies through multiple serial mixture of experts with adaptive weight allocation. Extensive experiments demonstrate that our method significantly surpasses the state-of-the-art methods, while greatly reducing parameters and computational load. Our code will be released on https://github.com/uchiha99999/ESG-Net.