D3G: Exploring Gaussian Prior for Temporal Sentence Grounding with Glance Annotation

TL;DR

This paper introduces D3G, a novel weakly supervised framework for temporal sentence grounding that uses glance annotations and Gaussian priors to effectively locate video moments with reduced annotation effort.

Contribution

The paper proposes a Dynamic Gaussian prior based Grounding framework with Glance annotation (D3G), combining semantic alignment and dynamic distribution adjustment for improved weakly supervised TSG.

Findings

Outperforms state-of-the-art weakly supervised methods significantly.

Narrows the performance gap with fully supervised approaches.

Proves effectiveness across three challenging benchmarks.

Abstract

Temporal sentence grounding (TSG) aims to locate a specific moment from an untrimmed video with a given natural language query. Recently, weakly supervised methods still have a large performance gap compared to fully supervised ones, while the latter requires laborious timestamp annotations. In this study, we aim to reduce the annotation cost yet keep competitive performance for TSG task compared to fully supervised ones. To achieve this goal, we investigate a recently proposed glance-supervised temporal sentence grounding task, which requires only single frame annotation (referred to as glance annotation) for each query. Under this setup, we propose a Dynamic Gaussian prior based Grounding framework with Glance annotation (D3G), which consists of a Semantic Alignment Group Contrastive Learning module (SA-GCL) and a Dynamic Gaussian prior Adjustment module (DGA). Specifically, SA-GCL samples reliable positive moments from a 2D temporal map via jointly leveraging Gaussian prior and semantic consistency, which contributes to aligning the positive sentence-moment pairs in the joint embedding space. Moreover, to alleviate the annotation bias resulting from glance annotation and model complex queries consisting of multiple events, we propose the DGA module, which adjusts the distribution dynamically to approximate the ground truth of target moments. Extensive experiments on three challenging benchmarks verify the effectiveness of the proposed D3G. It outperforms the state-of-the-art weakly supervised methods by a large margin and narrows the performance gap compared to fully supervised methods. Code is available at https://github.com/solicucu/D3G.