Weakly Supervised Temporal Sentence Grounding via Positive Sample Mining

TL;DR

This paper introduces a Positive Sample Mining framework for weakly supervised temporal sentence grounding, improving discrimination by mining and leveraging similar positive samples during training.

Contribution

It proposes a novel PSM framework with contrastive and rank losses to better utilize similar samples, enhancing weakly supervised grounding performance.

Findings

Outperforms existing methods on WSTSG and VideoQA tasks.

Effectively leverages semantically similar samples for training.

Demonstrates improved accuracy and robustness in temporal grounding.

Abstract

The task of weakly supervised temporal sentence grounding (WSTSG) aims to detect temporal intervals corresponding to a language description from untrimmed videos with only video-level video-language correspondence. For an anchor sample, most existing approaches generate negative samples either from other videos or within the same video for contrastive learning. However, some training samples are highly similar to the anchor sample, directly regarding them as negative samples leads to difficulties for optimization and ignores the correlations between these similar samples and the anchor sample. To address this, we propose Positive Sample Mining (PSM), a novel framework that mines positive samples from the training set to provide more discriminative supervision. Specifically, for a given anchor sample, we partition the remaining training set into semantically similar and dissimilar subsets based on the similarity of their text queries. To effectively leverage these correlations, we introduce a PSM-guided contrastive loss to ensure that the anchor proposal is closer to similar samples and further from dissimilar ones. Additionally, we design a PSM-guided rank loss to ensure that similar samples are closer to the anchor proposal than to the negative intra-video proposal, aiming to distinguish the anchor proposal and the negative intra-video proposal. Experiments on the WSTSG and grounded VideoQA tasks demonstrate the effectiveness and superiority of our method.