Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer

TL;DR

Pix2Vex introduces a neural network that reconstructs 3D geometry from images using a novel smooth differentiable renderer and image translation, enabling training with minimal supervision and diverse image types.

Contribution

The paper presents a new smooth differentiable renderer and a training scheme that allows 3D reconstruction from varied images without requiring 3D ground truth.

Findings

Achieves 3D reconstruction with minimal supervision.

Uses a smooth $DR$ to avoid discontinuities at occlusions.

Enables training on diverse image types.

Abstract

The long-coveted task of reconstructing 3D geometry from images is still a standing problem. In this paper, we build on the power of neural networks and introduce Pix2Vex, a network trained to convert camera-captured images into 3D geometry. We present a novel differentiable renderer ($DR$) as a forward validation means during training. Our key insight is that $DR$s produce images of a particular appearance, different from typical input images. Hence, we propose adding an image-to-image translation component, converting between these rendering styles. This translation closes the training loop, while allowing to use minimal supervision only, without needing any 3D model as ground truth. Unlike state-of-the-art methods, our $DR$ is $C^\infty$ smooth and thus does not display any discontinuities at occlusions or dis-occlusions. Through our novel training scheme, our network can train on different types of images, where previous work can typically only train on images of a similar appearance to those rendered by a $DR$.