ArrowGEV: Grounding Events in Video via Learning the Arrow of Time

TL;DR

ArrowGEV is a reinforcement learning framework that enhances video event grounding by explicitly modeling temporal directionality, inspired by the arrow of time, leading to better accuracy and understanding.

Contribution

It introduces a novel approach to incorporate temporal directionality into vision-language models for improved event grounding and reasoning in videos.

Findings

Improves grounding precision in videos.

Enhances recognition of temporal directionality.

Boosts overall video understanding and reasoning.

Abstract

Grounding events in videos serves as a fundamental capability in video analysis. While Vision Language Models (VLMs) are increasingly employed for this task, existing approaches predominantly train models to associate events with timestamps in the forward video only. This paradigm hinders VLMs from capturing the inherent temporal structure and directionality of events, thereby limiting robustness and generalization. To address this limitation, inspired by the arrow of time in physics, which characterizes the intrinsic directionality of temporal processes, we propose ArrowGEV, a reinforcement learning framework that explicitly models temporal directionality in events to improve both event grounding and temporal directionality understanding in VLMs. Specifically, we categorize events into time-sensitive (e.g., putting down a bag) and time-insensitive (e.g., holding a towel in the left hand). The former denote events whose reversal substantially alters their meaning, while the latter remain semantically unchanged under reversal. For time-sensitive events, ArrowGEV introduces a reward that encourages VLMs to discriminate between forward and backward videos, whereas for time-insensitive events, it enforces consistent grounding across both directions. Extensive experiments demonstrate that ArrowGEV not only improves grounding precision and temporal directionality recognition, but also enhances general video understanding and reasoning ability.