CLIP-AE: CLIP-assisted Cross-view Audio-Visual Enhancement for Unsupervised Temporal Action Localization

TL;DR

This paper introduces CLIP-AE, a novel unsupervised method for temporal action localization that leverages cross-view audio-visual information and visual language pre-training to improve boundary detection without requiring annotations.

Contribution

The paper proposes a CLIP-assisted cross-view audiovisual enhancement framework that combines visual language pre-training and audio perception for improved unsupervised temporal action localization.

Findings

Outperforms state-of-the-art methods on public datasets

Effectively utilizes audio cues for boundary detection

Reduces reliance on highly discriminative region focus

Abstract

Temporal Action Localization (TAL) has garnered significant attention in information retrieval. Existing supervised or weakly supervised methods heavily rely on labeled temporal boundaries and action categories, which are labor-intensive and time-consuming. Consequently, unsupervised temporal action localization (UTAL) has gained popularity. However, current methods face two main challenges: 1) Classification pre-trained features overly focus on highly discriminative regions; 2) Solely relying on visual modality information makes it difficult to determine contextual boundaries. To address these issues, we propose a CLIP-assisted cross-view audiovisual enhanced UTAL method. Specifically, we introduce visual language pre-training (VLP) and classification pre-training-based collaborative enhancement to avoid excessive focus on highly discriminative regions; we also incorporate audio perception to provide richer contextual boundary information. Finally, we introduce a self-supervised cross-view learning paradigm to achieve multi-view perceptual enhancement without additional annotations. Extensive experiments on two public datasets demonstrate our model's superiority over several state-of-the-art competitors.