Identity-aware Graph Memory Network for Action Detection

TL;DR

This paper introduces an identity-aware graph memory network that models actor identities in both long-term and short-term contexts to improve action detection accuracy in videos.

Contribution

The paper proposes a novel hierarchical graph neural network and dual attention module to explicitly incorporate actor identity information into action detection models.

Findings

Achieves state-of-the-art results on AVA v2.1 and v2.2 datasets.

Effectively models long-term actor relations and reduces interference from different identities.

Demonstrates significant improvement over existing methods in action detection accuracy.

Abstract

Action detection plays an important role in high-level video understanding and media interpretation. Many existing studies fulfill this spatio-temporal localization by modeling the context, capturing the relationship of actors, objects, and scenes conveyed in the video. However, they often universally treat all the actors without considering the consistency and distinctness between individuals, leaving much room for improvement. In this paper, we explicitly highlight the identity information of the actors in terms of both long-term and short-term context through a graph memory network, namely identity-aware graph memory network (IGMN). Specifically, we propose the hierarchical graph neural network (HGNN) to comprehensively conduct long-term relation modeling within the same identity as well as between different ones. Regarding short-term context, we develop a dual attention module (DAM) to generate identity-aware constraint to reduce the influence of interference by the actors of different identities. Extensive experiments on the challenging AVA dataset demonstrate the effectiveness of our method, which achieves state-of-the-art results on AVA v2.1 and v2.2.