Universal Video Temporal Grounding with Generative Multi-modal Large Language Models

TL;DR

This paper introduces UniTime, a universal video temporal grounding model leveraging generative multi-modal large language models, capable of accurately localizing video segments based on natural language queries across diverse video types and lengths.

Contribution

We propose a novel approach that uses strong MLLMs for temporal grounding, incorporating temporal information and adaptive frame scaling for robustness across various video durations.

Findings

Outperforms state-of-the-art methods in multiple benchmarks.

Effective in zero-shot and finetuned settings.

Enhances VideoQA accuracy when used as a moment retriever.

Abstract

This paper presents a computational model for universal video temporal grounding, which accurately localizes temporal moments in videos based on natural language queries (e.g., questions or descriptions). Unlike existing methods that are often limited to specific video domains or durations, we propose UniTime, a robust and universal video grounding model leveraging the strong vision-language understanding capabilities of generative Multi-modal Large Language Models (MLLMs). Our model effectively handles videos of diverse views, genres, and lengths while comprehending complex language queries. The key contributions include: (i) We consider steering strong MLLMs for temporal grounding in videos. To enable precise timestamp outputs, we incorporate temporal information by interleaving timestamp tokens with video tokens. (ii) By training the model to handle videos with different input granularities through adaptive frame scaling, our approach achieves robust temporal grounding for both short and long videos. (iii) Comprehensive experiments show that UniTime outperforms state-of-the-art approaches in both zero-shot and dataset-specific finetuned settings across five public temporal grounding benchmarks. (iv) When employed as a preliminary moment retriever for long-form video question-answering (VideoQA), UniTime significantly improves VideoQA accuracy, highlighting its value for complex video understanding tasks.