Estimation of Reliable Proposal Quality for Temporal Action Detection

TL;DR

This paper introduces a novel approach for temporal action detection that aligns classification and localization tasks by estimating proposal quality from moment and region perspectives, leading to state-of-the-art results.

Contribution

It proposes Boundary Evaluate Module and Region Evaluate Module to improve proposal quality estimation, addressing temporal misalignment in anchor-free TAD methods.

Findings

Improves THUMOS14 mAP by 3.6% to 63.6%.

Enhances ActivityNet-1.3 mAP to 36.2%.

Modules are compatible with existing frameworks.

Abstract

Temporal action detection (TAD) aims to locate and recognize the actions in an untrimmed video. Anchor-free methods have made remarkable progress which mainly formulate TAD into two tasks: classification and localization using two separate branches. This paper reveals the temporal misalignment between the two tasks hindering further progress. To address this, we propose a new method that gives insights into moment and region perspectives simultaneously to align the two tasks by acquiring reliable proposal quality. For the moment perspective, Boundary Evaluate Module (BEM) is designed which focuses on local appearance and motion evolvement to estimate boundary quality and adopts a multi-scale manner to deal with varied action durations. For the region perspective, we introduce Region Evaluate Module (REM) which uses a new and efficient sampling method for proposal feature representation containing more contextual information compared with point feature to refine category score and proposal boundary. The proposed Boundary Evaluate Module and Region Evaluate Module (BREM) are generic, and they can be easily integrated with other anchor-free TAD methods to achieve superior performance. In our experiments, BREM is combined with two different frameworks and improves the performance on THUMOS14 by 3.6% and 1.0% respectively, reaching a new state-of-the-art (63.6% average mAP). Meanwhile, a competitive result of 36.2% average mAP is achieved on ActivityNet-1.3 with the consistent improvement of BREM. The codes are released at https://github.com/Junshan233/BREM.