MM-Pyramid: Multimodal Pyramid Attentional Network for Audio-Visual Event Localization and Video Parsing

TL;DR

The paper introduces MM-Pyramid, a multimodal pyramid attentional network that captures multi-scale semantic features for improved audio-visual event localization and video parsing.

Contribution

It proposes a novel attentive feature pyramid module and an adaptive semantic fusion module for multi-scale event localization in videos.

Findings

Effective in localizing events of different lengths

Outperforms previous methods on audio-visual event localization

Improves weakly-supervised video parsing accuracy

Abstract

Recognizing and localizing events in videos is a fundamental task for video understanding. Since events may occur in auditory and visual modalities, multimodal detailed perception is essential for complete scene comprehension. Most previous works attempted to analyze videos from a holistic perspective. However, they do not consider semantic information at multiple scales, which makes the model difficult to localize events in different lengths. In this paper, we present a Multimodal Pyramid Attentional Network (\textbf{MM-Pyramid}) for event localization. Specifically, we first propose the attentive feature pyramid module. This module captures temporal pyramid features via several stacking pyramid units, each of them is composed of a fixed-size attention block and dilated convolution block. We also design an adaptive semantic fusion module, which leverages a unit-level attention block and a selective fusion block to integrate pyramid features interactively. Extensive experiments on audio-visual event localization and weakly-supervised audio-visual video parsing tasks verify the effectiveness of our approach.