Self-supervised Pre-training with Masked Shape Prediction for 3D Scene Understanding

TL;DR

This paper introduces Masked Shape Prediction (MSP), a novel self-supervised pre-training framework for 3D scene understanding that leverages geometric shape cues to improve feature learning and downstream task performance.

Contribution

It proposes a new masked shape prediction framework with context-enhanced shape targets and a specialized architecture to prevent shape leakage, advancing 3D scene understanding.

Findings

MSP improves performance on multiple 3D understanding tasks.

MSP effectively captures geometric shape cues for better feature representations.

Pre-training with MSP boosts downstream task accuracy.

Abstract

Masked signal modeling has greatly advanced self-supervised pre-training for language and 2D images. However, it is still not fully explored in 3D scene understanding. Thus, this paper introduces Masked Shape Prediction (MSP), a new framework to conduct masked signal modeling in 3D scenes. MSP uses the essential 3D semantic cue, i.e., geometric shape, as the prediction target for masked points. The context-enhanced shape target consisting of explicit shape context and implicit deep shape feature is proposed to facilitate exploiting contextual cues in shape prediction. Meanwhile, the pre-training architecture in MSP is carefully designed to alleviate the masked shape leakage from point coordinates. Experiments on multiple 3D understanding tasks on both indoor and outdoor datasets demonstrate the effectiveness of MSP in learning good feature representations to consistently boost downstream performance.