Temporal Action Localization with Cross Layer Task Decoupling and Refinement

TL;DR

This paper introduces a novel approach for temporal action localization that effectively disentangles classification and localization tasks using cross layer feature decoupling and refinement, leading to state-of-the-art results.

Contribution

The proposed CLTDR method integrates multi-layer features for better task decoupling and refinement, and introduces the lightweight GMG module for multi-granularity feature extraction.

Findings

Achieves state-of-the-art performance on five benchmarks.

Effectively disentangles classification and localization tasks.

Improves feature utilization with the GMG module.

Abstract

Temporal action localization (TAL) involves dual tasks to classify and localize actions within untrimmed videos. However, the two tasks often have conflicting requirements for features. Existing methods typically employ separate heads for classification and localization tasks but share the same input feature, leading to suboptimal performance. To address this issue, we propose a novel TAL method with Cross Layer Task Decoupling and Refinement (CLTDR). Based on the feature pyramid of video, CLTDR strategy integrates semantically strong features from higher pyramid layers and detailed boundary-aware boundary features from lower pyramid layers to effectively disentangle the action classification and localization tasks. Moreover, the multiple features from cross layers are also employed to refine and align the disentangled classification and regression results. At last, a lightweight Gated Multi-Granularity (GMG) module is proposed to comprehensively extract and aggregate video features at instant, local, and global temporal granularities. Benefiting from the CLTDR and GMG modules, our method achieves state-of-the-art performance on five challenging benchmarks: THUMOS14, MultiTHUMOS, EPIC-KITCHENS-100, ActivityNet-1.3, and HACS. Our code and pre-trained models are publicly available at: https://github.com/LiQiang0307/CLTDR-GMG.