GraphThinker: Reinforcing Temporally Grounded Video Reasoning with Event Graph Thinking

TL;DR

GraphThinker enhances video reasoning by explicitly modeling event relations and reinforcing visual grounding, significantly reducing hallucinations and improving localization and question-answering accuracy.

Contribution

It introduces a structured event representation and visual attention reward for reinforcement finetuning of multimodal models, addressing weak grounding and hallucination issues.

Findings

Over 4% improvement in IoU=0.3 for moment localization on RexTime.

9.8% reduction in temporal sequence hallucination on VidHalluc.

7.6% improvement in Binary QA accuracy by reducing action hallucination.

Abstract

Video reasoning requires a fine-grained understanding of the temporal dependencies and event-level relations between objects and events in videos. Current Multimodal Large Language Models (MLLMs) are prone to severe temporal hallucinations in video reasoning. An underlying cause of these hallucinations is weak visual-temporal grounding and the lack of explicit structure for modelling event relations. Models often rely on auxiliary text, such as dense captions, rather than explicitly anchoring their reasoning in actual visual evidence. However, these textual representations are inherently unstructured and fail to provide explicit causal constraints needed to guide the model's reasoning. In this work, we propose GraphThinker, a reinforcement finetuning method that constructs a structured event representation of a video and enforces visual grounding to jointly reduce reasoning hallucinations. Specifically, we employ an MLLM to construct an Event-based Video Scene Graph (EVSG) that captures both intra- and inter-event relations, guiding a structured video reasoning process. Moreover, we address the weak grounding issue by introducing a novel visual attention reward during reinforcement finetuning that encourages the model to actively attend to reliable visual cues. On the RexTime dataset, GraphThinker achieves an over 4% improvement in IoU=0.3 for moment localisation. On the VidHalluc dataset, GraphThinker achieves a 9.8% improvement in reducing temporal sequence hallucination and a 7.6% gain in Binary QA in reducing action hallucination, compared to the state-of-the-art methods.