Datasets and Recipes for Video Temporal Grounding via Reinforcement Learning

TL;DR

This paper presents a novel two-stage training framework combining supervised fine-tuning and reinforcement learning to enhance the accuracy and robustness of video temporal grounding models, especially in challenging scenarios.

Contribution

It introduces a new training approach that leverages high-quality cold start data and difficulty-controlled RL, improving temporal localization and reasoning in VTG models.

Findings

Outperforms existing VTG models on multiple benchmarks.

Enhances model robustness in open-domain scenarios.

Highlights the importance of dataset quality and training strategies.

Abstract

Video Temporal Grounding (VTG) aims to localize relevant temporal segments in videos given natural language queries. Despite recent progress with large vision-language models (LVLMs) and instruction-tuning, existing approaches often suffer from limited temporal awareness and poor generalization. In this work, we introduce a two-stage training framework that integrates supervised fine-tuning with reinforcement learning (RL) to improve both the accuracy and robustness of VTG models. Our approach first leverages high-quality curated cold start data for SFT initialization, followed by difficulty-controlled RL to further enhance temporal localization and reasoning abilities. Comprehensive experiments on multiple VTG benchmarks demonstrate that our method consistently outperforms existing models, particularly in challenging and open-domain scenarios. We conduct an in-depth analysis of training strategies and dataset curation, highlighting the importance of both high-quality cold start data and difficulty-controlled RL. To facilitate further research and industrial adoption, we release all intermediate datasets, models, and code to the community.