EAR: Enhancing Uni-Modal Representations for Weakly Supervised Audio-Visual Video Parsing

TL;DR

This paper introduces a novel framework that enhances uni-modal representations to improve weakly supervised audio-visual video parsing, addressing the limitations of existing multi-modal focused strategies.

Contribution

It proposes a similarity-based label migration and a soft-constrained approach to better preserve uni-modal semantics during video parsing.

Findings

Outperforms state-of-the-art methods in pseudo-label accuracy.

Achieves superior localization of audio, visual, and audio-visual events.

Enhances the understanding of uni-modal events for better video parsing.

Abstract

Weakly supervised Audio-Visual Video Parsing (AVVP) aims to recognize and temporally localize audio, visual, and audio-visual events in videos using only coarse-grained labels. Faced with the challenging task settings, existing research advances along two main paths: pre-training pseudo-label generators for fine-grained cross-modal semantic guidance, or refining AVVP model architectures to enhance audio-visual fusion. However, since audio and visual signals are typically unaligned, achieving accurate video parsing fundamentally relies on precise perception of uni-modal events. Yet these multi-modal focused strategies excessively emphasize multi-modal fusion while inadequately guiding and preserving uni-modal semantics, resulting in noisy pseudo-labels and sub-optimal video parsing performance. This paper proposes a novel framework that enhances uni-modal representations for both the pseudo-label generator and the AVVP model. Specifically, we introduce a similarity-based label migration approach to annotate pre-training data, thereby enabling the pseudo-label generator to better understand uni-modal events. We also employ a soft-constrained manner to refine modeling of uni-modal features in parallel with multi-modal fusion. These designs enable coordinated attention to both uni-modal and cross-modal representations, thus boosting the localization performance for events. Extensive experiments show that our method outperforms state-of-the-art methods in both pseudo-label and AVVP performance.