STS-Mixer: Spatio-Temporal-Spectral Mixer for 4D Point Cloud Video Understanding

TL;DR

STS-Mixer introduces a spectral domain approach to 4D point cloud video understanding, decomposing signals into frequency bands to better capture geometric and dynamic scene information.

Contribution

It proposes a novel spectral analysis framework and a unified spatio-temporal-spectral mixer for improved 4D point cloud video understanding.

Findings

Achieves superior performance on 3D action recognition benchmarks.

Outperforms existing methods in 4D semantic segmentation.

Effectively captures geometric details through spectral decomposition.

Abstract

4D point cloud videos capture rich spatial and temporal dynamics of scenes which possess unique values in various 4D understanding tasks. However, most existing methods work in the spatiotemporal domain where the underlying geometric characteristics of 4D point cloud videos are hard to capture, leading to degraded representation learning and understanding of 4D point cloud videos. We address the above challenge from a complementary spectral perspective. By transforming 4D point cloud videos into graph spectral signals, we can decompose them into multiple frequency bands each of which captures distinct geometric structures of point cloud videos. Our spectral analysis reveals that the decomposed low-frequency signals capture more coarse shapes while high-frequency signals encode more fine-grained geometry details. Building on these observations, we design Spatio-Temporal-Spectral Mixer (STS-Mixer), a unified framework that mixes spatial, temporal, and spectral representations of point cloud videos. STS-Mixer integrates multi-band delineated spectral signals with spatiotemporal information to capture rich geometries and temporal dynamics, while enabling fine-grained and holistic understanding of 4D point cloud videos. Extensive experiments show that STS-Mixer achieves superior performance consistently across multiple widely adopted benchmarks on both 3D action recognition and 4D semantic segmentation tasks. Code and models are available at https://github.com/Vegetebird/STS-Mixer.