UST-SSM: Unified Spatio-Temporal State Space Models for Point Cloud Video Modeling

TL;DR

The paper introduces UST-SSM, a novel model for point cloud video analysis that reorganizes unordered points into semantic sequences and enhances spatio-temporal feature aggregation, improving action recognition accuracy.

Contribution

The paper proposes UST-SSM, extending state space models with spatial-temporal selection, structure aggregation, and interaction sampling for better point cloud video modeling.

Findings

Effective on multiple datasets including MSR-Action3D, NTU RGB+D, and Synthia 4D.

Improves spatio-temporal feature utilization and action recognition accuracy.

Code available at provided GitHub link.

Abstract

Point cloud videos capture dynamic 3D motion while reducing the effects of lighting and viewpoint variations, making them highly effective for recognizing subtle and continuous human actions. Although Selective State Space Models (SSMs) have shown good performance in sequence modeling with linear complexity, the spatio-temporal disorder of point cloud videos hinders their unidirectional modeling when directly unfolding the point cloud video into a 1D sequence through temporally sequential scanning. To address this challenge, we propose the Unified Spatio-Temporal State Space Model (UST-SSM), which extends the latest advancements in SSMs to point cloud videos. Specifically, we introduce Spatial-Temporal Selection Scanning (STSS), which reorganizes unordered points into semantic-aware sequences through prompt-guided clustering, thereby enabling the effective utilization of points that are spatially and temporally distant yet similar within the sequence. For missing 4D geometric and motion details, Spatio-Temporal Structure Aggregation (STSA) aggregates spatio-temporal features and compensates. To improve temporal interaction within the sampled sequence, Temporal Interaction Sampling (TIS) enhances fine-grained temporal dependencies through non-anchor frame utilization and expanded receptive fields. Experimental results on the MSR-Action3D, NTU RGB+D, and Synthia 4D datasets validate the effectiveness of our method. Our code is available at https://github.com/wangzy01/UST-SSM.