Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer

TL;DR

This paper introduces DIB-R, a differentiable rendering framework that enables gradient-based optimization of 3D object properties from 2D images, advancing 3D understanding in machine learning.

Contribution

The paper presents a novel interpolation-based differentiable renderer that allows analytical gradients for all pixels, facilitating end-to-end learning of 3D object properties from 2D supervision.

Findings

Effective optimization of vertex positions, colors, normals, and textures.

Successful application to 3D object prediction and textured object generation.

Operates solely with 2D supervision in experiments.

Abstract

Many machine learning models operate on images, but ignore the fact that images are 2D projections formed by 3D geometry interacting with light, in a process called rendering. Enabling ML models to understand image formation might be key for generalization. However, due to an essential rasterization step involving discrete assignment operations, rendering pipelines are non-differentiable and thus largely inaccessible to gradient-based ML techniques. In this paper, we present {\emph DIB-R}, a differentiable rendering framework which allows gradients to be analytically computed for all pixels in an image. Key to our approach is to view foreground rasterization as a weighted interpolation of local properties and background rasterization as a distance-based aggregation of global geometry. Our approach allows for accurate optimization over vertex positions, colors, normals, light directions and texture coordinates through a variety of lighting models. We showcase our approach in two ML applications: single-image 3D object prediction, and 3D textured object generation, both trained using exclusively using 2D supervision. Our project website is: https://nv-tlabs.github.io/DIB-R/