High Fidelity Semantic Shape Completion for Point Clouds using Latent Optimization

TL;DR

This paper introduces a novel learning-based method for semantic shape completion of point clouds using latent optimization, achieving high-fidelity reconstructions of incomplete 3D shapes without database retrieval.

Contribution

It presents a new approach combining autoencoders and GANs with latent manifold optimization for direct point cloud completion, improving over existing methods.

Findings

Successfully reconstructs large missing regions in point clouds

Achieves high fidelity in shape completion without database retrieval

Outperforms previous methods in qualitative and quantitative evaluations

Abstract

Semantic shape completion is a challenging problem in 3D computer vision where the task is to generate a complete 3D shape using a partial 3D shape as input. We propose a learning-based approach to complete incomplete 3D shapes through generative modeling and latent manifold optimization. Our algorithm works directly on point clouds. We use an autoencoder and a GAN to learn a distribution of embeddings for point clouds of object classes. An input point cloud with missing regions is first encoded to a feature vector. The representations learnt by the GAN are then used to find the best latent vector on the manifold using a combined optimization that finds a vector in the manifold of plausible vectors that is close to the original input (both in the feature space and the output space of the decoder). Experiments show that our algorithm is capable of successfully reconstructing point clouds with large missing regions with very high fidelity without having to rely on exemplar based database retrieval.