Spatio-Temporal Action Detection Under Large Motion

TL;DR

This paper improves spatiotemporal action detection in videos with large motion by tracking actors and aggregating features along tracks, significantly enhancing detection accuracy especially for fast-moving actions.

Contribution

It introduces a track-aware feature aggregation method that outperforms traditional cuboid-based approaches in large motion scenarios.

Findings

Track-aware aggregation improves detection accuracy for large motion actions.

Proposed method achieves state-of-the-art results on MultiSports dataset.

Enhances feature representation under large actor motion and deformation.

Abstract

Current methods for spatiotemporal action tube detection often extend a bounding box proposal at a given keyframe into a 3D temporal cuboid and pool features from nearby frames. However, such pooling fails to accumulate meaningful spatiotemporal features if the position or shape of the actor shows large 2D motion and variability through the frames, due to large camera motion, large actor shape deformation, fast actor action and so on. In this work, we aim to study the performance of cuboid-aware feature aggregation in action detection under large action. Further, we propose to enhance actor feature representation under large motion by tracking actors and performing temporal feature aggregation along the respective tracks. We define the actor motion with intersection-over-union (IoU) between the boxes of action tubes/tracks at various fixed time scales. The action having a large motion would result in lower IoU over time, and slower actions would maintain higher IoU. We find that track-aware feature aggregation consistently achieves a large improvement in action detection performance, especially for actions under large motion compared to the cuboid-aware baseline. As a result, we also report state-of-the-art on the large-scale MultiSports dataset. The Code is available at https://github.com/gurkirt/ActionTrackDetectron.