Using Adaptive Gradient for Texture Learning in Single-View 3D Reconstruction

TL;DR

This paper introduces a novel gradient-based sampling algorithm and a FID-based loss to improve texture generation in single-view 3D reconstruction, achieving better shape and texture quality without 3D supervision.

Contribution

It proposes a new sampling method optimizing gradient of predicted coordinates and incorporates FID loss to enhance texture quality in unsupervised 3D reconstruction.

Findings

Significantly improved texture quality in 3D models.

Outperformed previous methods in shape and texture accuracy.

Achieved better results without 3D supervision.

Abstract

Recently, learning-based approaches for 3D model reconstruction have attracted attention owing to its modern applications such as Extended Reality(XR), robotics and self-driving cars. Several approaches presented good performance on reconstructing 3D shapes by learning solely from images, i.e., without using 3D models in training. Challenges, however, remain in texture generation due to the gap between 2D and 3D modals. In previous work, the grid sampling mechanism from Spatial Transformer Networks was adopted to sample color from an input image to formulate texture. Despite its success, the existing framework has limitations on searching scope in sampling, resulting in flaws in generated texture and consequentially on rendered 3D models. In this paper, to solve that issue, we present a novel sampling algorithm by optimizing the gradient of predicted coordinates based on the variance on the sampling image. Taking into account the semantics of the image, we adopt Frechet Inception Distance (FID) to form a loss function in learning, which helps bridging the gap between rendered images and input images. As a result, we greatly improve generated texture. Furthermore, to optimize 3D shape reconstruction and to accelerate convergence at training, we adopt part segmentation and template learning in our model. Without any 3D supervision in learning, and with only a collection of single-view 2D images, the shape and texture learned by our model outperform those from previous work. We demonstrate the performance with experimental results on a publically available dataset.