RTime-QA: A Benchmark for Atomic Temporal Event Understanding in Large Multi-modal Models

TL;DR

This paper introduces RTime-QA, a challenging benchmark for evaluating large multi-modal models' understanding of atomic temporal events in videos, and proposes RTime-IT for improving their temporal comprehension through instruction tuning.

Contribution

The paper presents a new benchmark RTime-QA for atomic temporal event understanding and an instruction-tuning dataset RTime-IT to enhance LMMs' temporal reasoning capabilities.

Findings

Qwen2-VL scores only 34.6 on RTime-QA, showing the benchmark's difficulty.

Fine-tuning on RTime-IT improves Qwen2-VL's score to 65.9.

RTime-QA reveals significant gaps in current LMMs' temporal understanding.

Abstract

Understanding accurate atomic temporal event is essential for video comprehension. However, current video-language benchmarks often fall short to evaluate Large Multi-modal Models' (LMMs) temporal event understanding capabilities, as they can be effectively addressed using image-language models. In this paper, we introduce RTime-QA, a novel benchmark specifically designed to assess the atomic temporal event understanding ability of LMMs. RTime-QA comprises 822 high-quality, carefully-curated video-text questions, each meticulously annotated by human experts. Each question features a video depicting an atomic temporal event, paired with both correct answers and temporal negative descriptions, specifically designed to evaluate temporal understanding. To advance LMMs' temporal event understanding ability, we further introduce RTime-IT, a 14k instruction-tuning dataset that employs a similar annotation process as RTime-QA. Extensive experimental analysis demonstrates that RTime-QA presents a significant challenge for LMMs: the state-of-the-art model Qwen2-VL achieves only 34.6 on strict-ACC metric, substantially lagging behind human performance. Furthermore, our experiments reveal that RTime-IT effectively enhance LMMs' capacity in temporal understanding. By fine-tuning on RTime-IT, our Qwen2-VL achieves 65.9 on RTime-QA.