Shape Inpainting using 3D Generative Adversarial Network and Recurrent Convolutional Networks

TL;DR

This paper introduces a hybrid 3D shape inpainting framework combining a 3D-ED-GAN for global structure and an LRCN for high-resolution details, overcoming GPU memory limitations.

Contribution

The paper presents a novel hybrid approach integrating 3D-ED-GAN and LRCN for high-resolution 3D shape inpainting, addressing memory constraints and improving detail preservation.

Findings

Produces high-resolution 3D reconstructions from corrupted models

Effectively captures global and local shape features

Outperforms existing methods on real-world and synthetic data

Abstract

Recent advances in convolutional neural networks have shown promising results in 3D shape completion. But due to GPU memory limitations, these methods can only produce low-resolution outputs. To inpaint 3D models with semantic plausibility and contextual details, we introduce a hybrid framework that combines a 3D Encoder-Decoder Generative Adversarial Network (3D-ED-GAN) and a Long-term Recurrent Convolutional Network (LRCN). The 3D-ED-GAN is a 3D convolutional neural network trained with a generative adversarial paradigm to fill missing 3D data in low-resolution. LRCN adopts a recurrent neural network architecture to minimize GPU memory usage and incorporates an Encoder-Decoder pair into a Long Short-term Memory Network. By handling the 3D model as a sequence of 2D slices, LRCN transforms a coarse 3D shape into a more complete and higher resolution volume. While 3D-ED-GAN captures global contextual structure of the 3D shape, LRCN localizes the fine-grained details. Experimental results on both real-world and synthetic data show reconstructions from corrupted models result in complete and high-resolution 3D objects.