RCL: Recurrent Continuous Localization for Temporal Action Detection

TL;DR

This paper introduces Recurrent Continuous Localization (RCL), a novel approach for temporal action detection that employs a fully continuous, differentiable anchoring scheme, improving detection accuracy over discretized methods.

Contribution

The paper proposes a continuous, fully differentiable anchoring representation for temporal action detection, enabling better detection of segments with arbitrary length and seamless integration into existing detectors.

Findings

RCL outperforms discretized methods by ~2% mAP.

Achieves 52.92% mAP@0.5 on THUMOS14.

Achieves 37.65% mAP on ActivityNet v1.3.

Abstract

Temporal representation is the cornerstone of modern action detection techniques. State-of-the-art methods mostly rely on a dense anchoring scheme, where anchors are sampled uniformly over the temporal domain with a discretized grid, and then regress the accurate boundaries. In this paper, we revisit this foundational stage and introduce Recurrent Continuous Localization (RCL), which learns a fully continuous anchoring representation. Specifically, the proposed representation builds upon an explicit model conditioned with video embeddings and temporal coordinates, which ensure the capability of detecting segments with arbitrary length. To optimize the continuous representation, we develop an effective scale-invariant sampling strategy and recurrently refine the prediction in subsequent iterations. Our continuous anchoring scheme is fully differentiable, allowing to be seamlessly integrated into existing detectors, e.g., BMN and G-TAD. Extensive experiments on two benchmarks demonstrate that our continuous representation steadily surpasses other discretized counterparts by ~2% mAP. As a result, RCL achieves 52.92% mAP@0.5 on THUMOS14 and 37.65% mAP on ActivtiyNet v1.3, outperforming all existing single-model detectors.