Boundary-sensitive Pre-training for Temporal Localization in Videos

TL;DR

This paper introduces a boundary-sensitive pre-training method for temporal video localization that synthesizes boundary annotations from existing datasets, significantly improving transferability and achieving state-of-the-art results.

Contribution

The paper proposes a novel boundary-sensitive pretext task that synthesizes temporal boundaries for pre-training, enhancing transferability to localization tasks without needing manual boundary annotations.

Findings

BSP outperforms existing pre-training methods.

BSP achieves state-of-the-art results on multiple datasets.

Synthesizing boundaries improves temporal localization accuracy.

Abstract

Many video analysis tasks require temporal localization thus detection of content changes. However, most existing models developed for these tasks are pre-trained on general video action classification tasks. This is because large scale annotation of temporal boundaries in untrimmed videos is expensive. Therefore no suitable datasets exist for temporal boundary-sensitive pre-training. In this paper for the first time, we investigate model pre-training for temporal localization by introducing a novel boundary-sensitive pretext (BSP) task. Instead of relying on costly manual annotations of temporal boundaries, we propose to synthesize temporal boundaries in existing video action classification datasets. With the synthesized boundaries, BSP can be simply conducted via classifying the boundary types. This enables the learning of video representations that are much more transferable to downstream temporal localization tasks. Extensive experiments show that the proposed BSP is superior and complementary to the existing action classification based pre-training counterpart, and achieves new state-of-the-art performance on several temporal localization tasks.