Leveraging Single-View Images for Unsupervised 3D Point Cloud Completion

TL;DR

This paper introduces Cross-PCC, an unsupervised 3D point cloud completion method that leverages 2D images instead of requiring complete 3D data, achieving superior results compared to existing unsupervised approaches.

Contribution

The novel approach uses 2D images for supervision in point cloud completion, eliminating the need for 3D complete point clouds and outperforming previous unsupervised methods.

Findings

Outperforms state-of-the-art unsupervised methods

Achieves comparable results to some supervised methods

First to use only 2D images for unsupervised point cloud completion

Abstract

Point clouds captured by scanning devices are often incomplete due to occlusion. To overcome this limitation, point cloud completion methods have been developed to predict the complete shape of an object based on its partial input. These methods can be broadly classified as supervised or unsupervised. However, both categories require a large number of 3D complete point clouds, which may be difficult to capture. In this paper, we propose Cross-PCC, an unsupervised point cloud completion method without requiring any 3D complete point clouds. We only utilize 2D images of the complete objects, which are easier to capture than 3D complete and clean point clouds. Specifically, to take advantage of the complementary information from 2D images, we use a single-view RGB image to extract 2D features and design a fusion module to fuse the 2D and 3D features extracted from the partial point cloud. To guide the shape of predicted point clouds, we project the predicted points of the object to the 2D plane and use the foreground pixels of its silhouette maps to constrain the position of the projected points. To reduce the outliers of the predicted point clouds, we propose a view calibrator to move the points projected to the background into the foreground by the single-view silhouette image. To the best of our knowledge, our approach is the first point cloud completion method that does not require any 3D supervision. The experimental results of our method are superior to those of the state-of-the-art unsupervised methods by a large margin. Moreover, our method even achieves comparable performance to some supervised methods. We will make the source code publicly available at https://github.com/ltwu6/cross-pcc.