Unleashing Hour-Scale Video Training for Long Video-Language Understanding

TL;DR

This paper introduces VideoMarathon, a large-scale dataset of hour-long videos with QA pairs, and Hour-LLaVA, a new Video-LMM that effectively models long videos for improved understanding and benchmark performance.

Contribution

The paper presents a novel dataset for hour-long videos and a new model that enables efficient hour-scale video-language training and inference.

Findings

Hour-LLaVA achieves state-of-the-art results on long video benchmarks.

VideoMarathon significantly extends training video durations and task diversity.

The model effectively leverages memory augmentation for long-term video comprehension.

Abstract

Recent long-form video-language understanding benchmarks have driven progress in video large multimodal models (Video-LMMs). However, the scarcity of well-annotated long videos has left the training of hour-long Video-LMMs underexplored. To close this gap, we present VideoMarathon, a large-scale hour-long video instruction-following dataset. This dataset includes around 9,700 hours of long videos sourced from diverse domains, ranging from 3 to 60 minutes per video. Specifically, it contains 3.3M high-quality QA pairs, spanning six fundamental topics: temporality, spatiality, object, action, scene, and event. Compared to existing video instruction datasets, VideoMarathon significantly extends training video durations up to 1 hour, and supports 22 diverse tasks requiring both short- and long-term video comprehension. Building on VideoMarathon, we propose Hour-LLaVA, a powerful and efficient Video-LMM for hour-scale video-language modeling. It enables hour-long video training and inference at 1-FPS sampling by leveraging a memory augmentation module, which adaptively integrates question-relevant and spatiotemporally informative semantics from the cached full video context. In our experiments, Hour-LLaVA achieves the best performance on multiple representative long video-language benchmarks, demonstrating the high quality of the VideoMarathon dataset and the superiority of the Hour-LLaVA model.