CTRN: Class-Temporal Relational Network for Action Detection

TL;DR

This paper introduces CTRN, an end-to-end network that models class and temporal relations to improve action detection in densely labeled untrimmed videos, addressing challenges like composite and co-occurring actions.

Contribution

The paper presents a novel Class-Temporal Relational Network with modules for feature filtering, relation modeling, and co-occurring action detection, advancing state-of-the-art performance.

Findings

Achieves state-of-the-art results on three challenging datasets.

Effectively models class and temporal relations for dense action detection.

Improves detection of co-occurring actions using privileged knowledge.

Abstract

Action detection is an essential and challenging task, especially for densely labelled datasets of untrimmed videos. There are many real-world challenges in those datasets, such as composite action, co-occurring action, and high temporal variation of instance duration. For handling these challenges, we propose to explore both the class and temporal relations of detected actions. In this work, we introduce an end-to-end network: Class-Temporal Relational Network (CTRN). It contains three key components: (1) The Representation Transform Module filters the class-specific features from the mixed representations to build graph-structured data. (2) The Class-Temporal Module models the class and temporal relations in a sequential manner. (3) G-classifier leverages the privileged knowledge of the snippet-wise co-occurring action pairs to further improve the co-occurring action detection. We evaluate CTRN on three challenging densely labelled datasets and achieve state-of-the-art performance, reflecting the effectiveness and robustness of our method.