DyFADet: Dynamic Feature Aggregation for Temporal Action Detection

TL;DR

DyFADet introduces a dynamic feature aggregation approach with adaptive kernels and receptive fields, significantly improving temporal action detection across diverse video datasets.

Contribution

The paper proposes a novel dynamic feature aggregation module and a dynamic TAD head, enabling better modeling of actions with varying lengths and complexities.

Findings

Achieves state-of-the-art performance on multiple TAD benchmarks.

Effectively models actions with diverse temporal ranges.

Demonstrates the benefits of dynamic receptive fields in video analysis.

Abstract

Recent proposed neural network-based Temporal Action Detection (TAD) models are inherently limited to extracting the discriminative representations and modeling action instances with various lengths from complex scenes by shared-weights detection heads. Inspired by the successes in dynamic neural networks, in this paper, we build a novel dynamic feature aggregation (DFA) module that can simultaneously adapt kernel weights and receptive fields at different timestamps. Based on DFA, the proposed dynamic encoder layer aggregates the temporal features within the action time ranges and guarantees the discriminability of the extracted representations. Moreover, using DFA helps to develop a Dynamic TAD head (DyHead), which adaptively aggregates the multi-scale features with adjusted parameters and learned receptive fields better to detect the action instances with diverse ranges from videos. With the proposed encoder layer and DyHead, a new dynamic TAD model, DyFADet, achieves promising performance on a series of challenging TAD benchmarks, including HACS-Segment, THUMOS14, ActivityNet-1.3, Epic-Kitchen 100, Ego4D-Moment QueriesV1.0, and FineAction. Code is released to https://github.com/yangle15/DyFADet-pytorch.