Efficient Action Detection in Untrimmed Videos via Multi-Task Learning

TL;DR

This paper presents a multi-task learning framework for efficient action detection in untrimmed videos, jointly performing proposal, recognition, and localization refinement, outperforming state-of-the-art methods in accuracy and speed.

Contribution

It introduces a novel multi-task approach with a temporal actionness regression module and random shear augmentation, improving efficiency and accuracy over existing sequential methods.

Findings

Outperforms state-of-the-art accuracy

Runs several times faster than sequential models

Does not require dense trajectory features

Abstract

This paper studies the joint learning of action recognition and temporal localization in long, untrimmed videos. We employ a multi-task learning framework that performs the three highly related steps of action proposal, action recognition, and action localization refinement in parallel instead of the standard sequential pipeline that performs the steps in order. We develop a novel temporal actionness regression module that estimates what proportion of a clip contains action. We use it for temporal localization but it could have other applications like video retrieval, surveillance, summarization, etc. We also introduce random shear augmentation during training to simulate viewpoint change. We evaluate our framework on three popular video benchmarks. Results demonstrate that our joint model is efficient in terms of storage and computation in that we do not need to compute and cache dense trajectory features, and that it is several times faster than its sequential ConvNets counterpart. Yet, despite being more efficient, it outperforms state-of-the-art methods with respect to accuracy.