STMixer: A One-Stage Sparse Action Detector

TL;DR

STMixer is an innovative one-stage sparse action detector that uses adaptive feature sampling and dual-branch feature mixing to efficiently and accurately detect actions in videos, outperforming existing methods.

Contribution

The paper introduces STMixer, a novel end-to-end action detection framework with adaptive feature sampling and dynamic feature mixing, improving performance and convergence speed.

Findings

Achieves state-of-the-art results on AVA, UCF101-24, and JHMDB datasets.

Demonstrates superior performance over traditional two-stage detectors.

Provides an efficient, end-to-end solution for video action detection.

Abstract

Traditional video action detectors typically adopt the two-stage pipeline, where a person detector is first employed to generate actor boxes and then 3D RoIAlign is used to extract actor-specific features for classification. This detection paradigm requires multi-stage training and inference, and cannot capture context information outside the bounding box. Recently, a few query-based action detectors are proposed to predict action instances in an end-to-end manner. However, they still lack adaptability in feature sampling and decoding, thus suffering from the issues of inferior performance or slower convergence. In this paper, we propose a new one-stage sparse action detector, termed STMixer. STMixer is based on two core designs. First, we present a query-based adaptive feature sampling module, which endows our STMixer with the flexibility of mining a set of discriminative features from the entire spatiotemporal domain. Second, we devise a dual-branch feature mixing module, which allows our STMixer to dynamically attend to and mix video features along the spatial and the temporal dimension respectively for better feature decoding. Coupling these two designs with a video backbone yields an efficient end-to-end action detector. Without bells and whistles, our STMixer obtains the state-of-the-art results on the datasets of AVA, UCF101-24, and JHMDB.