PointCG: Self-supervised Point Cloud Learning via Joint Completion and Generation

TL;DR

PointCG introduces a self-supervised learning framework for 3D point clouds by combining shape completion and image generation tasks, improving the encoder's understanding of geometric structures.

Contribution

It integrates masked point modeling and 3D-to-2D generation into a unified pre-training framework with novel HPC and AIG modules for enhanced 3D feature learning.

Findings

Outperforms baseline methods in downstream tasks

Effective in capturing geometric details

Demonstrates robustness across various datasets

Abstract

The core of self-supervised point cloud learning lies in setting up appropriate pretext tasks, to construct a pre-training framework that enables the encoder to perceive 3D objects effectively. In this paper, we integrate two prevalent methods, masked point modeling (MPM) and 3D-to-2D generation, as pretext tasks within a pre-training framework. We leverage the spatial awareness and precise supervision offered by these two methods to address their respective limitations: ambiguous supervision signals and insensitivity to geometric information. Specifically, the proposed framework, abbreviated as PointCG, consists of a Hidden Point Completion (HPC) module and an Arbitrary-view Image Generation (AIG) module. We first capture visible points from arbitrary views as inputs by removing hidden points. Then, HPC extracts representations of the inputs with an encoder and completes the entire shape with a decoder, while AIG is used to generate rendered images based on the visible points' representations. Extensive experiments demonstrate the superiority of the proposed method over the baselines in various downstream tasks. Our code will be made available upon acceptance.