Neural 3D Mesh Renderer

TL;DR

This paper introduces a neural 3D mesh renderer with an approximate gradient for rasterization, enabling end-to-end training for 3D mesh reconstruction and editing from 2D images, outperforming voxel-based methods.

Contribution

It proposes a differentiable rasterization technique allowing neural networks to learn 3D meshes directly from 2D supervision, which was not feasible before.

Findings

Outperforms voxel-based approaches in 3D reconstruction

Enables gradient-based 3D mesh editing from 2D supervision

Demonstrates applications like style transfer and DeepDream on 3D meshes

Abstract

For modeling the 3D world behind 2D images, which 3D representation is most appropriate? A polygon mesh is a promising candidate for its compactness and geometric properties. However, it is not straightforward to model a polygon mesh from 2D images using neural networks because the conversion from a mesh to an image, or rendering, involves a discrete operation called rasterization, which prevents back-propagation. Therefore, in this work, we propose an approximate gradient for rasterization that enables the integration of rendering into neural networks. Using this renderer, we perform single-image 3D mesh reconstruction with silhouette image supervision and our system outperforms the existing voxel-based approach. Additionally, we perform gradient-based 3D mesh editing operations, such as 2D-to-3D style transfer and 3D DeepDream, with 2D supervision for the first time. These applications demonstrate the potential of the integration of a mesh renderer into neural networks and the effectiveness of our proposed renderer.