Boundary-Recovering Network for Temporal Action Detection

TL;DR

The paper introduces the Boundary-Recovering Network (BRN), which enhances temporal action detection by addressing boundary ambiguity and the vanishing boundary problem through a novel scale-time feature construction and exchange mechanism.

Contribution

BRN introduces a new scale-time feature construction and exchange method to recover ambiguous action boundaries in temporal detection tasks.

Findings

Outperforms state-of-the-art on ActivityNet-v1.3 and THUMOS14

Reduces the vanishing boundary problem significantly

Improves boundary localization accuracy

Abstract

Temporal action detection (TAD) is challenging, yet fundamental for real-world video applications. Large temporal scale variation of actions is one of the most primary difficulties in TAD. Naturally, multi-scale features have potential in localizing actions of diverse lengths as widely used in object detection. Nevertheless, unlike objects in images, actions have more ambiguity in their boundaries. That is, small neighboring objects are not considered as a large one while short adjoining actions can be misunderstood as a long one. In the coarse-to-fine feature pyramid via pooling, these vague action boundaries can fade out, which we call 'vanishing boundary problem'. To this end, we propose Boundary-Recovering Network (BRN) to address the vanishing boundary problem. BRN constructs scale-time features by introducing a new axis called scale dimension by interpolating multi-scale features to the same temporal length. On top of scale-time features, scale-time blocks learn to exchange features across scale levels, which can effectively settle down the issue. Our extensive experiments demonstrate that our model outperforms the state-of-the-art on the two challenging benchmarks, ActivityNet-v1.3 and THUMOS14, with remarkably reduced degree of the vanishing boundary problem.