Computer Vision based group activity detection and action spotting

TL;DR

This paper introduces a computer vision framework that combines deep learning and graph-based reasoning to detect group activities and individual actions in multi-person scenes, improving accuracy in complex scenarios.

Contribution

It presents a novel integration of Mask R-CNN, feature fusion, and Actor Relation Graphs with GCNs for enhanced group activity recognition and action spotting.

Findings

Improved recognition accuracy on the Collective Activity dataset.

Effective modeling of human interactions using graph neural networks.

Robust performance in both crowded and non-crowded scenes.

Abstract

Group activity detection in multi-person scenes is challenging due to complex human interactions, occlusions, and variations in appearance over time. This work presents a computer vision based framework for group activity recognition and action spotting using a combination of deep learning models and graph based relational reasoning. The system first applies Mask R-CNN to obtain accurate actor localization through bounding boxes and instance masks. Multiple backbone networks, including Inception V3, MobileNet, and VGG16, are used to extract feature maps, and RoIAlign is applied to preserve spatial alignment when generating actor specific features. The mask information is then fused with the feature maps to obtain refined masked feature representations for each actor. To model interactions between individuals, we construct Actor Relation Graphs that encode appearance similarity and positional relations using methods such as normalized cross correlation, sum of absolute differences, and dot product. Graph Convolutional Networks operate on these graphs to reason about relationships and predict both individual actions and group level activities. Experiments on the Collective Activity dataset demonstrate that the combination of mask based feature refinement, robust similarity search, and graph neural network reasoning leads to improved recognition performance across both crowded and non crowded scenarios. This approach highlights the potential of integrating segmentation, feature extraction, and relational graph reasoning for complex video understanding tasks.