TimeRefine: Temporal Grounding with Time Refining Video LLM

TL;DR

TimeRefine introduces a progressive temporal refining approach for video grounding with LLMs, significantly improving localization accuracy by iterative offset prediction and auxiliary supervision.

Contribution

It reformulates temporal grounding as a multi-step refinement process and adds an auxiliary loss to enhance temporal perception in Video LLMs.

Findings

Achieves 3.6% mIoU improvement on ActivityNet

Achieves 5.0% mIoU improvement on Charades-STA

Demonstrates effective integration into existing LLM-based methods

Abstract

Video temporal grounding aims to localize relevant temporal boundaries in a video given a textual prompt. Recent work has focused on enabling Video LLMs to perform video temporal grounding via next-token prediction of temporal timestamps. However, accurately localizing timestamps in videos remains challenging for Video LLMs when relying solely on temporal token prediction. Our proposed TimeRefine addresses this challenge in two ways. First, instead of directly predicting the start and end timestamps, we reformulate the temporal grounding task as a temporal refining task: the model first makes rough predictions and then refines them by predicting offsets to the target segment. This refining process is repeated multiple times, through which the model progressively self-improves its temporal localization accuracy. Second, to enhance the model's temporal perception capabilities, we incorporate an auxiliary prediction head that penalizes the model more if a predicted segment deviates further from the ground truth, thus encouraging the model to make closer and more accurate predictions. Our plug-and-play method can be integrated into most LLM-based temporal grounding approaches. The experimental results demonstrate that TimeRefine achieves 3.6% and 5.0% mIoU improvements on the ActivityNet and Charades-STA datasets, respectively. Code and pretrained models will be released.