LSTC: Boosting Atomic Action Detection with Long-Short-Term Context

TL;DR

This paper introduces a novel framework that decomposes atomic action detection into short-term and long-term context analysis, significantly improving performance by leveraging dense cues and high-order interactions.

Contribution

It proposes a new Long-Short-Term Context (LSTC) approach that independently models and combines short-term and long-term temporal dependencies for atomic action detection.

Findings

Both temporal grains improve detection accuracy.

Significant performance gains on benchmark datasets.

The approach outperforms previous state-of-the-art methods.

Abstract

In this paper, we place the atomic action detection problem into a Long-Short Term Context (LSTC) to analyze how the temporal reliance among video signals affect the action detection results. To do this, we decompose the action recognition pipeline into short-term and long-term reliance, in terms of the hypothesis that the two kinds of context are conditionally independent given the objective action instance. Within our design, a local aggregation branch is utilized to gather dense and informative short-term cues, while a high order long-term inference branch is designed to reason the objective action class from high-order interaction between actor and other person or person pairs. Both branches independently predict the context-specific actions and the results are merged in the end. We demonstrate that both temporal grains are beneficial to atomic action recognition. On the mainstream benchmarks of atomic action detection, our design can bring significant performance gain from the existing state-of-the-art pipeline. The code of this project can be found at [this url](https://github.com/TencentYoutuResearch/ActionDetection-LSTC)