Bridging the Gap: A Unified Video Comprehension Framework for Moment Retrieval and Highlight Detection

TL;DR

This paper introduces UVCOM, a unified framework that effectively addresses both Video Moment Retrieval and Highlight Detection by integrating local and global video understanding through multi-aspect contrastive learning.

Contribution

The paper proposes a novel unified framework, UVCOM, that jointly tackles MR and HD with task-specific design and multi-granularity integration, outperforming existing methods.

Findings

UVCOM outperforms state-of-the-art methods on multiple datasets.

Multi-aspect contrastive learning enhances local and global video understanding.

Task-specific design improves the effectiveness of joint MR and HD.

Abstract

Video Moment Retrieval (MR) and Highlight Detection (HD) have attracted significant attention due to the growing demand for video analysis. Recent approaches treat MR and HD as similar video grounding problems and address them together with transformer-based architecture. However, we observe that the emphasis of MR and HD differs, with one necessitating the perception of local relationships and the other prioritizing the understanding of global contexts. Consequently, the lack of task-specific design will inevitably lead to limitations in associating the intrinsic specialty of two tasks. To tackle the issue, we propose a Unified Video COMprehension framework (UVCOM) to bridge the gap and jointly solve MR and HD effectively. By performing progressive integration on intra and inter-modality across multi-granularity, UVCOM achieves the comprehensive understanding in processing a video. Moreover, we present multi-aspect contrastive learning to consolidate the local relation modeling and global knowledge accumulation via well aligned multi-modal space. Extensive experiments on QVHighlights, Charades-STA, TACoS , YouTube Highlights and TVSum datasets demonstrate the effectiveness and rationality of UVCOM which outperforms the state-of-the-art methods by a remarkable margin.