Semi-Supervised Temporal Action Detection with Proposal-Free Masking

TL;DR

This paper introduces SPOT, a proposal-free semi-supervised method for temporal action detection that reduces error propagation and improves performance by parallel localization and classification with interaction mechanisms.

Contribution

The paper proposes a novel proposal-free architecture for semi-supervised temporal action detection that eliminates localization-classification error propagation and enhances prediction accuracy.

Findings

SPOT outperforms state-of-the-art methods on standard benchmarks.

The parallel localization and classification architecture reduces error propagation.

Interaction mechanisms improve prediction refinement.

Abstract

Existing temporal action detection (TAD) methods rely on a large number of training data with segment-level annotations. Collecting and annotating such a training set is thus highly expensive and unscalable. Semi-supervised TAD (SS-TAD) alleviates this problem by leveraging unlabeled videos freely available at scale. However, SS-TAD is also a much more challenging problem than supervised TAD, and consequently much under-studied. Prior SS-TAD methods directly combine an existing proposal-based TAD method and a SSL method. Due to their sequential localization (e.g, proposal generation) and classification design, they are prone to proposal error propagation. To overcome this limitation, in this work we propose a novel Semi-supervised Temporal action detection model based on PropOsal-free Temporal mask (SPOT) with a parallel localization (mask generation) and classification architecture. Such a novel design effectively eliminates the dependence between localization and classification by cutting off the route for error propagation in-between. We further introduce an interaction mechanism between classification and localization for prediction refinement, and a new pretext task for self-supervised model pre-training. Extensive experiments on two standard benchmarks show that our SPOT outperforms state-of-the-art alternatives, often by a large margin. The PyTorch implementation of SPOT is available at https://github.com/sauradip/SPOT