Seeing the Arrow of Time in Large Multimodal Models

TL;DR

This paper introduces ArrowRL, a reinforcement learning strategy that enhances large multimodal models' ability to understand the arrow of time in videos, significantly improving temporal comprehension and question answering accuracy.

Contribution

The paper proposes ArrowRL, a novel RL-based training method with reverse rewards, and introduces AoTBench, a new benchmark for evaluating temporal understanding in video models.

Findings

ArrowRL improves temporal perception in LMMs.

Significant accuracy gains on AoTBench and VQA benchmarks.

Highlights the importance of AoT understanding in video models.

Abstract

The Arrow of Time (AoT)-time's irreversible flow shaping physical events-is fundamental to video comprehension, yet remains a significant challenge for modern large multimodal models (LMMs). Current LMMs struggle to perceive and utilize temporal directionality in video when responding to language queries, obstructing deeper temporal understanding. We tackle this deficiency by first providing a critical analysis of existing benchmarks and models. We then introduce ArrowRL, a reinforcement learning (RL)-based training strategy with an innovative reverse reward that instills AoT awareness by encouraging divergent video interpretations between forward and reversed visual frames. For rigorous evaluation, we additionally develop AoTBench, a new multi-faceted benchmark probing temporally challenging questions. Experiments show ArrowRL greatly advances temporal perception: it not only achieves substantial improvements on our challenging AoTBench but also demonstrably boosts performance on standard video question answering (VQA) benchmarks (with peak accuracy gains reaching over 20% and 10% respectively). This validates ArrowRL's effectiveness and highlights the critical need for dedicated AoT understanding in LMMs.