Towards Open-Vocabulary Audio-Visual Event Localization

TL;DR

This paper introduces the open-vocabulary audio-visual event localization task, a new dataset, and baseline methods to recognize and localize both seen and unseen events in videos, advancing the field beyond closed-set limitations.

Contribution

The paper proposes the OV-AVEBench dataset and evaluation metrics, establishing a new open-vocabulary AVEL task with baseline approaches using pretrained multimodal features.

Findings

Baseline methods demonstrate the feasibility of open-vocabulary event localization.

The dataset enables evaluation of models on both seen and unseen event categories.

Pretrained multimodal features improve zero-shot and few-shot localization performance.

Abstract

The Audio-Visual Event Localization (AVEL) task aims to temporally locate and classify video events that are both audible and visible. Most research in this field assumes a closed-set setting, which restricts these models' ability to handle test data containing event categories absent (unseen) during training. Recently, a few studies have explored AVEL in an open-set setting, enabling the recognition of unseen events as ``unknown'', but without providing category-specific semantics. In this paper, we advance the field by introducing the Open-Vocabulary Audio-Visual Event Localization (OV-AVEL) problem, which requires localizing audio-visual events and predicting explicit categories for both seen and unseen data at inference. To address this new task, we propose the OV-AVEBench dataset, comprising 24,800 videos across 67 real-life audio-visual scenes (seen:unseen = 46:21), each with manual segment-level annotation. We also establish three evaluation metrics for this task. Moreover, we investigate two baseline approaches, one training-free and one using a further fine-tuning paradigm. Specifically, we utilize the unified multimodal space from the pretrained ImageBind model to extract audio, visual, and textual (event classes) features. The training-free baseline then determines predictions by comparing the consistency of audio-text and visual-text feature similarities. The fine-tuning baseline incorporates lightweight temporal layers to encode temporal relations within the audio and visual modalities, using OV-AVEBench training data for model fine-tuning. We evaluate these baselines on the proposed OV-AVEBench dataset and discuss potential directions for future work in this new field.