Recurrent Space-time Graph Neural Networks

TL;DR

This paper introduces a recurrent space-time graph neural network that effectively captures complex spatio-temporal interactions in visual data, outperforming existing methods in activity recognition tasks.

Contribution

The paper presents a novel neural graph model that processes space and time recurrently, integrating local features and high-level interactions for improved spatio-temporal understanding.

Findings

Outperforms strong baselines in activity recognition

Achieves state-of-the-art results on Something-Something dataset

Demonstrates effectiveness through extensive experiments and ablation studies

Abstract

Learning in the space-time domain remains a very challenging problem in machine learning and computer vision. Current computational models for understanding spatio-temporal visual data are heavily rooted in the classical single-image based paradigm. It is not yet well understood how to integrate information in space and time into a single, general model. We propose a neural graph model, recurrent in space and time, suitable for capturing both the local appearance and the complex higher-level interactions of different entities and objects within the changing world scene. Nodes and edges in our graph have dedicated neural networks for processing information. Nodes operate over features extracted from local parts in space and time and previous memory states. Edges process messages between connected nodes at different locations and spatial scales or between past and present time. Messages are passed iteratively in order to transmit information globally and establish long range interactions. Our model is general and could learn to recognize a variety of high level spatio-temporal concepts and be applied to different learning tasks. We demonstrate, through extensive experiments and ablation studies, that our model outperforms strong baselines and top published methods on recognizing complex activities in video. Moreover, we obtain state-of-the-art performance on the challenging Something-Something human-object interaction dataset.