Spatiotemporal Deformable Scene Graphs for Complex Activity Detection

TL;DR

This paper introduces a novel spatiotemporal deformable scene graph approach for complex activity detection, combining action tube detection, deformable geometry modeling, and graph convolutional networks, applicable across domains like autonomous driving and surgery.

Contribution

The paper presents a new deformable, spatiotemporal scene graph framework with a 3D deformable RoI pooling layer, and provides new annotated datasets for autonomous driving and surgical actions.

Findings

Outperforms graph-based competitors on augmented datasets

Demonstrates adaptability across different domains

Introduces new annotations for complex activities

Abstract

Long-term complex activity recognition and localisation can be crucial for decision making in autonomous systems such as smart cars and surgical robots. Here we address the problem via a novel deformable, spatiotemporal scene graph approach, consisting of three main building blocks: (i) action tube detection, (ii) the modelling of the deformable geometry of parts, and (iii) a graph convolutional network. Firstly, action tubes are detected in a series of snippets. Next, a new 3D deformable RoI pooling layer is designed for learning the flexible, deformable geometry of the constituent action tubes. Finally, a scene graph is constructed by considering all parts as nodes and connecting them based on different semantics such as order of appearance, sharing the same action label and feature similarity. We also contribute fresh temporal complex activity annotation for the recently released ROAD autonomous driving and SARAS-ESAD surgical action datasets and show the adaptability of our framework to different domains. Our method is shown to significantly outperform graph-based competitors on both augmented datasets.