LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos

TL;DR

LongVALE introduces a comprehensive benchmark for time-aware, multi-modal understanding of long videos, integrating vision, audio, and language with fine-grained event annotations to advance omni-modal perception.

Contribution

It is the first benchmark with 105K multi-modal events and precise temporal boundaries, enabling detailed omni-modal event understanding and LLM-based temporal video analysis.

Findings

Effective multi-modal event detection and captioning

Enhanced video understanding with fine-grained temporal annotations

Potential for advancing omni-modal video perception

Abstract

Despite impressive advancements in video understanding, most efforts remain limited to coarse-grained or visual-only video tasks. However, real-world videos encompass omni-modal information (vision, audio, and speech) with a series of events forming a cohesive storyline. The lack of multi-modal video data with fine-grained event annotations and the high cost of manual labeling are major obstacles to comprehensive omni-modality video perception. To address this gap, we propose an automatic pipeline consisting of high-quality multi-modal video filtering, semantically coherent omni-modal event boundary detection, and cross-modal correlation-aware event captioning. In this way, we present LongVALE, the first-ever Vision-Audio-Language Event understanding benchmark comprising 105K omni-modal events with precise temporal boundaries and detailed relation-aware captions within 8.4K high-quality long videos. Further, we build a baseline that leverages LongVALE to enable video large language models (LLMs) for omni-modality fine-grained temporal video understanding for the first time. Extensive experiments demonstrate the effectiveness and great potential of LongVALE in advancing comprehensive multi-modal video understanding.