Exploiting Auxiliary Caption for Video Grounding

TL;DR

This paper introduces ACNet, a novel framework that leverages auxiliary captions generated through dense captioning and contrastive learning to improve video grounding accuracy by addressing annotation sparsity.

Contribution

The paper proposes a new auxiliary caption-based approach with CGA and ACCL to enhance video grounding, outperforming existing methods on multiple datasets.

Findings

Significant performance improvement over state-of-the-art methods

Effective use of dense captioning and contrastive learning

Robust across multiple public datasets

Abstract

Video grounding aims to locate a moment of interest matching the given query sentence from an untrimmed video. Previous works ignore the {sparsity dilemma} in video annotations, which fails to provide the context information between potential events and query sentences in the dataset. In this paper, we contend that exploiting easily available captions which describe general actions, i.e., auxiliary captions defined in our paper, will significantly boost the performance. To this end, we propose an Auxiliary Caption Network (ACNet) for video grounding. Specifically, we first introduce dense video captioning to generate dense captions and then obtain auxiliary captions by Non-Auxiliary Caption Suppression (NACS). To capture the potential information in auxiliary captions, we propose Caption Guided Attention (CGA) project the semantic relations between auxiliary captions and query sentences into temporal space and fuse them into visual representations. Considering the gap between auxiliary captions and ground truth, we propose Asymmetric Cross-modal Contrastive Learning (ACCL) for constructing more negative pairs to maximize cross-modal mutual information. Extensive experiments on three public datasets (i.e., ActivityNet Captions, TACoS and ActivityNet-CG) demonstrate that our method significantly outperforms state-of-the-art methods.