TubeR: Tubelet Transformer for Video Action Detection

TL;DR

TubeR introduces a novel end-to-end transformer-based approach for video action detection that models spatio-temporal dynamics directly, outperforming previous methods on standard benchmarks.

Contribution

It presents a simple, unified model that detects action tubelets without relying on actor detectors or hand-crafted proposals, using tubelet-queries and attention mechanisms.

Findings

Outperforms previous state-of-the-art on AVA, UCF101-24, JHMDB51-21 datasets.

Effectively models dynamic spatio-temporal features with tubelet-attention.

Maintains good performance on long video clips.

Abstract

We propose TubeR: a simple solution for spatio-temporal video action detection. Different from existing methods that depend on either an off-line actor detector or hand-designed actor-positional hypotheses like proposals or anchors, we propose to directly detect an action tubelet in a video by simultaneously performing action localization and recognition from a single representation. TubeR learns a set of tubelet-queries and utilizes a tubelet-attention module to model the dynamic spatio-temporal nature of a video clip, which effectively reinforces the model capacity compared to using actor-positional hypotheses in the spatio-temporal space. For videos containing transitional states or scene changes, we propose a context aware classification head to utilize short-term and long-term context to strengthen action classification, and an action switch regression head for detecting the precise temporal action extent. TubeR directly produces action tubelets with variable lengths and even maintains good results for long video clips. TubeR outperforms the previous state-of-the-art on commonly used action detection datasets AVA, UCF101-24 and JHMDB51-21.