Towards Temporal Compositional Reasoning in Long-Form Sports Videos

TL;DR

This paper introduces SportsTime, a large-scale benchmark for long-form sports video understanding, and proposes Chain-of-Time Reasoning (CoTR), a method that improves temporal reasoning and evidence grounding in multimodal models.

Contribution

The paper presents a new benchmark dataset SportsTime and a novel reasoning method CoTR that enhances temporal compositional reasoning in sports videos.

Findings

CoTR improves temporal reasoning accuracy over baselines.

SportsTime enables better evaluation of long-horizon reasoning in sports videos.

CoTR enhances step-wise evidence grounding quality.

Abstract

Sports videos are a challenging domain for multimodal understanding because they involve complex and dynamic human activities. Despite rapid progress in Multimodal Large Language Models (MLLMs), long-horizon reasoning in sports videos remains difficult, as answering questions requires both locating temporally sparse evidence and integrating it into reasoning. We attribute this limitation to two closely coupled factors: insufficient supervision over temporally dispersed evidence, and the lack of methods that require models to identify, localize, and justify temporal evidence. To address these gaps, we introduce SportsTime, a large-scale benchmark for long-form sports video understanding, comprising 14K+ open-ended QA pairs and 50K+ step-wise temporal evidence annotations. Building on SportsTime, we propose Chain-of-Time Reasoning (CoTR), which treats reasoning as a process of temporally grounded evidence composition. Specifically, during training, CoTR introduces a temporal-reward GRPO to encourage temporally grounded reasoning. During inference, it employs an anchor-observe-infer evidence-seeking loop to iteratively localize, verify, and compose temporal evidence before producing the final answer. Experiments demonstrate the usefulness of SportsTime as a benchmark and the effectiveness of CoTR, which consistently improves temporal compositional reasoning and step-wise grounding quality over strong MLLM baselines.