Tempo-R0: A Video-MLLM for Temporal Video Grounding through Efficient Temporal Sensing Reinforcement Learning

TL;DR

Tempo-R0 introduces an innovative Video-MLLM for temporal video grounding that leverages efficient temporal sensing and reinforcement learning to improve boundary detection and relevance understanding in videos.

Contribution

The paper presents a novel Video-MLLM architecture with specialized preprocessing and reinforcement learning techniques for enhanced temporal video grounding.

Findings

Achieves around 3.5% improvement over SOTA on QVHighlights benchmarks.

Employs Self-adaptive Attention Allocation for efficient attention use.

Utilizes PIR-GRPO for improved temporal reasoning.

Abstract

Temporal Video Grounding (TVG), which requires pinpointing relevant temporal segments from video based on language query, has always been a highly challenging task in the field of video understanding. Videos often have a larger volume of information and redundancy than texts or images. Models should present comprehensive understanding of the whole video to accurately retrieve query-relevant clips. We thus propose Tempo-R0: a Video Multimodal Large Language Model (Video-MLLM) for the temporal video grounding task via multimodal temporal sensing reinforcement. Specifically, during the preprocessing stage of our pipeline, we employ Self-adaptive Attention Allocation (SAA) method based on frame content variation to efficiently use the MLLM's limited attention. The Explicit Timestamp-modal Aligned (ETA) method is also utilized to strengthen our model's capability to perceive the boundaries of events in the video. In the fine-tuning part of our pipeline, we creatively apply Partial Irrelevance Refusing-based Group Relative Policy Optimization (PIR-GRPO) in TVG area to foster model's temporal reasoning from not only accepting relevant video-query pairs but also refusing irrelevant ones. Experiments demonstrate that our method accomplishes a notable advantage over SOTA solutions by around 3.5% on both the original QVHighlights testbench and its corrected version with more reasonable ground truth annotations.