Masked Spatio-Temporal Structure Prediction for Self-supervised Learning on Point Cloud Videos

TL;DR

MaST-Pre is a self-supervised learning framework for point cloud videos that captures spatio-temporal structures through masked point-tube reconstruction and motion prediction, reducing reliance on labeled data.

Contribution

It introduces a novel masked spatio-temporal structure prediction approach with two self-supervised tasks for point cloud video understanding.

Findings

Effective in capturing appearance and motion in point cloud videos

Improves performance on multiple benchmark datasets

Reduces dependence on annotated data

Abstract

Recently, the community has made tremendous progress in developing effective methods for point cloud video understanding that learn from massive amounts of labeled data. However, annotating point cloud videos is usually notoriously expensive. Moreover, training via one or only a few traditional tasks (e.g., classification) may be insufficient to learn subtle details of the spatio-temporal structure existing in point cloud videos. In this paper, we propose a Masked Spatio-Temporal Structure Prediction (MaST-Pre) method to capture the structure of point cloud videos without human annotations. MaST-Pre is based on spatio-temporal point-tube masking and consists of two self-supervised learning tasks. First, by reconstructing masked point tubes, our method is able to capture the appearance information of point cloud videos. Second, to learn motion, we propose a temporal cardinality difference prediction task that estimates the change in the number of points within a point tube. In this way, MaST-Pre is forced to model the spatial and temporal structure in point cloud videos. Extensive experiments on MSRAction-3D, NTU-RGBD, NvGesture, and SHREC'17 demonstrate the effectiveness of the proposed method.