HLV-1K: A Large-scale Hour-Long Video Benchmark for Time-Specific Long Video Understanding

TL;DR

HLV-1K is a comprehensive large-scale benchmark dataset designed to evaluate models' ability to understand hour-long videos through diverse question-answering tasks, addressing a significant gap in long video understanding research.

Contribution

The paper introduces HLV-1K, the first large-scale, hour-long video dataset with extensive annotations for evaluating long video understanding models.

Findings

Existing models are tested on HLV-1K, revealing challenges in long-term video comprehension.

HLV-1K enables detailed evaluation of models across multiple reasoning levels.

The benchmark promotes development of more effective long video understanding methods.

Abstract

Multimodal large language models have become a popular topic in deep visual understanding due to many promising real-world applications. However, hour-long video understanding, spanning over one hour and containing tens of thousands of visual frames, remains under-explored because of 1) challenging long-term video analyses, 2) inefficient large-model approaches, and 3) lack of large-scale benchmark datasets. Among them, in this paper, we focus on building a large-scale hour-long long video benchmark, HLV-1K, designed to evaluate long video understanding models. HLV-1K comprises 1009 hour-long videos with 14,847 high-quality question answering (QA) and multi-choice question asnwering (MCQA) pairs with time-aware query and diverse annotations, covering frame-level, within-event-level, cross-event-level, and long-term reasoning tasks. We evaluate our benchmark using existing state-of-the-art methods and demonstrate its value for testing deep long video understanding capabilities at different levels and for various tasks. This includes promoting future long video understanding tasks at a granular level, such as deep understanding of long live videos, meeting recordings, and movies.