TimeExpert: An Expert-Guided Video LLM for Video Temporal Grounding

TL;DR

TimeExpert introduces a specialized Video Large Language Model that dynamically routes task-specific tokens to experts, significantly improving the accuracy and efficiency of video temporal grounding and related tasks.

Contribution

The paper presents a novel MoE-based Video-LLM that decomposes VTG tasks into sub-tasks with specialized routing, enhancing performance and computational efficiency.

Findings

Achieves state-of-the-art results on Dense Video Captioning.

Improves accuracy in Moment Retrieval and Video Highlight Detection.

Demonstrates effective task decomposition with MoE architecture.

Abstract

Video Temporal Grounding (VTG) aims to precisely identify video event segments in response to textual queries. The outputs of VTG tasks manifest as sequences of events, each defined by precise timestamps, saliency scores, and textual descriptions. Despite recent advances, a fundamental limitation persists in existing Video Large Language Models (Video-LLMs): they process all task tokens through identical and static pathways, failing to recognize that temporal localization, saliency assessment, and textual generation represent fundamentally distinct tasks requiring specialized processing. To address this, we introduce TimeExpert, a Mixture-of-Experts (MoE)-based Video-LLM that effectively decomposes VTG tasks by dynamically routing task-specific tokens (e.g., timestamps, saliency scores) to specialized experts, with increased computational efficiency. Our design choices enable precise handling of each subtask, leading to improved event modeling across diverse VTG applications. Extensive experiments demonstrate that TimeExpert consistently achieves state-of-the-art performance on various VTG tasks such as Dense Video Captioning, Moment Retrieval, and Video Highlight Detection.