Adaptive Proposal Generation Network for Temporal Sentence Localization in Videos

TL;DR

This paper introduces an Adaptive Proposal Generation Network (APGN) that combines the efficiency of bottom-up approaches with segment-level interaction, significantly improving temporal sentence localization in videos.

Contribution

The paper proposes a novel APGN that adaptively generates proposals by foreground-background classification, reducing redundancy and enhancing semantic quality, thus outperforming existing methods.

Findings

APGN achieves state-of-the-art results on three benchmarks.

The method reduces redundant proposals compared to traditional top-down approaches.

Semantic quality of proposals is significantly improved.

Abstract

We address the problem of temporal sentence localization in videos (TSLV). Traditional methods follow a top-down framework which localizes the target segment with pre-defined segment proposals. Although they have achieved decent performance, the proposals are handcrafted and redundant. Recently, bottom-up framework attracts increasing attention due to its superior efficiency. It directly predicts the probabilities for each frame as a boundary. However, the performance of bottom-up model is inferior to the top-down counterpart as it fails to exploit the segment-level interaction. In this paper, we propose an Adaptive Proposal Generation Network (APGN) to maintain the segment-level interaction while speeding up the efficiency. Specifically, we first perform a foreground-background classification upon the video and regress on the foreground frames to adaptively generate proposals. In this way, the handcrafted proposal design is discarded and the redundant proposals are decreased. Then, a proposal consolidation module is further developed to enhance the semantic of the generated proposals. Finally, we locate the target moments with these generated proposals following the top-down framework. Extensive experiments on three challenging benchmarks show that our proposed APGN significantly outperforms previous state-of-the-art methods.