Deep Learning-based Action Detection in Untrimmed Videos: A Survey

TL;DR

This survey comprehensively reviews deep learning methods for temporal and spatio-temporal action detection in untrimmed videos, covering various supervision levels, datasets, evaluation metrics, and real-world applications.

Contribution

It provides an extensive overview of recent algorithms, benchmarks, and challenges in deep learning-based action detection in untrimmed videos, highlighting future research directions.

Findings

Deep learning methods vary across supervision levels.

Benchmark datasets and metrics are standardized.

State-of-the-art methods show promising results.

Abstract

Understanding human behavior and activity facilitates advancement of numerous real-world applications, and is critical for video analysis. Despite the progress of action recognition algorithms in trimmed videos, the majority of real-world videos are lengthy and untrimmed with sparse segments of interest. The task of temporal activity detection in untrimmed videos aims to localize the temporal boundary of actions and classify the action categories. Temporal activity detection task has been investigated in full and limited supervision settings depending on the availability of action annotations. This paper provides an extensive overview of deep learning-based algorithms to tackle temporal action detection in untrimmed videos with different supervision levels including fully-supervised, weakly-supervised, unsupervised, self-supervised, and semi-supervised. In addition, this paper also reviews advances in spatio-temporal action detection where actions are localized in both temporal and spatial dimensions. Moreover, the commonly used action detection benchmark datasets and evaluation metrics are described, and the performance of the state-of-the-art methods are compared. Finally, real-world applications of temporal action detection in untrimmed videos and a set of future directions are discussed.