Skeleton-based Group Activity Recognition via Spatial-Temporal Panoramic Graph

TL;DR

This paper introduces a panoramic graph approach using multi-person skeletons and objects for group activity recognition, achieving state-of-the-art results with lower computational costs compared to RGB-based methods.

Contribution

The paper proposes a novel panoramic graph model that integrates skeletons and objects for efficient group activity recognition, eliminating the need for RGB data and complex interaction modules.

Findings

Outperforms RGB-based methods in accuracy and efficiency

Achieves state-of-the-art results on Volleyball and NBA datasets

Uses only 2D keypoints for activity prediction

Abstract

Group Activity Recognition aims to understand collective activities from videos. Existing solutions primarily rely on the RGB modality, which encounters challenges such as background variations, occlusions, motion blurs, and significant computational overhead. Meanwhile, current keypoint-based methods offer a lightweight and informative representation of human motions but necessitate accurate individual annotations and specialized interaction reasoning modules. To address these limitations, we design a panoramic graph that incorporates multi-person skeletons and objects to encapsulate group activity, offering an effective alternative to RGB video. This panoramic graph enables Graph Convolutional Network (GCN) to unify intra-person, inter-person, and person-object interactive modeling through spatial-temporal graph convolutions. In practice, we develop a novel pipeline that extracts skeleton coordinates using pose estimation and tracking algorithms and employ Multi-person Panoramic GCN (MP-GCN) to predict group activities. Extensive experiments on Volleyball and NBA datasets demonstrate that the MP-GCN achieves state-of-the-art performance in both accuracy and efficiency. Notably, our method outperforms RGB-based approaches by using only estimated 2D keypoints as input. Code is available at https://github.com/mgiant/MP-GCN