Single Shot Temporal Action Detection

TL;DR

This paper introduces a novel single shot temporal action detection network that directly detects actions in untrimmed videos, improving accuracy over existing methods by eliminating proposal generation.

Contribution

The paper proposes the SSAD network that directly detects actions using 1D temporal convolutions and explores architecture and feature fusion strategies for optimal performance.

Findings

SSAD outperforms state-of-the-art methods on THUMOS 2014 and MEXaction2 datasets.

Achieved significant mAP improvements at IoU 0.5.

Extensive experiments validate the effectiveness of the proposed approach.

Abstract

Temporal action detection is a very important yet challenging problem, since videos in real applications are usually long, untrimmed and contain multiple action instances. This problem requires not only recognizing action categories but also detecting start time and end time of each action instance. Many state-of-the-art methods adopt the "detection by classification" framework: first do proposal, and then classify proposals. The main drawback of this framework is that the boundaries of action instance proposals have been fixed during the classification step. To address this issue, we propose a novel Single Shot Action Detector (SSAD) network based on 1D temporal convolutional layers to skip the proposal generation step via directly detecting action instances in untrimmed video. On pursuit of designing a particular SSAD network that can work effectively for temporal action detection, we empirically search for the best network architecture of SSAD due to lacking existing models that can be directly adopted. Moreover, we investigate into input feature types and fusion strategies to further improve detection accuracy. We conduct extensive experiments on two challenging datasets: THUMOS 2014 and MEXaction2. When setting Intersection-over-Union threshold to 0.5 during evaluation, SSAD significantly outperforms other state-of-the-art systems by increasing mAP from 19.0% to 24.6% on THUMOS 2014 and from 7.4% to 11.0% on MEXaction2.