Harnessing Text-to-Image Diffusion Models for Point Cloud Self-Supervised Learning

TL;DR

This paper introduces PointSD, a novel framework that leverages large-scale text-to-image diffusion models like Stable Diffusion to improve 3D point cloud self-supervised learning, achieving enhanced semantic understanding.

Contribution

The paper proposes using pre-trained text-to-image diffusion models for 3D point cloud learning, bridging 2D generative models with 3D representation enhancement.

Findings

Improved performance on downstream point cloud tasks.

Effective alignment of 3D features with SD features.

Demonstrated benefits of large-scale diffusion models in 3D learning.

Abstract

Diffusion-based models, widely used in text-to-image generation, have proven effective in 2D representation learning. Recently, this framework has been extended to 3D self-supervised learning by constructing a conditional point generator for enhancing 3D representations. However, its performance remains constrained by the 3D diffusion model, which is trained on the available 3D datasets with limited size. We hypothesize that the robust capabilities of text-to-image diffusion models, particularly Stable Diffusion (SD), which is trained on large-scale datasets, can help overcome these limitations. To investigate this hypothesis, we propose PointSD, a framework that leverages the SD model for 3D self-supervised learning. By replacing the SD model's text encoder with a 3D encoder, we train a point-to-image diffusion model that allows point clouds to guide the denoising of rendered noisy images. With the trained point-to-image diffusion model, we use noise-free images as the input and point clouds as the condition to extract SD features. Next, we train a 3D backbone by aligning its features with these SD features, thereby facilitating direct semantic learning. Comprehensive experiments on downstream point cloud tasks and ablation studies demonstrate that the SD model can enhance point cloud self-supervised learning. Code is publicly available at https://github.com/wdttt/PointSD.