Inverse Transport Networks

TL;DR

This paper presents inverse transport networks, a learning architecture for inverse rendering that infers scene parameters from images using a differentiable renderer, improving generalization to unseen scenes.

Contribution

The introduction of inverse transport networks combined with a physically-accurate differentiable renderer for improved inverse rendering performance.

Findings

Efficient training of inverse transport networks using differentiable rendering.

Better generalization to unseen geometry and illumination compared to non-appearance-matching methods.

Demonstrated ability to infer physical scene parameters accurately.

Abstract

We introduce inverse transport networks as a learning architecture for inverse rendering problems where, given input image measurements, we seek to infer physical scene parameters such as shape, material, and illumination. During training, these networks are evaluated not only in terms of how close they can predict groundtruth parameters, but also in terms of whether the parameters they produce can be used, together with physically-accurate graphics renderers, to reproduce the input image measurements. To en- able training of inverse transport networks using stochastic gradient descent, we additionally create a general-purpose, physically-accurate differentiable renderer, which can be used to estimate derivatives of images with respect to arbitrary physical scene parameters. Our experiments demonstrate that inverse transport networks can be trained efficiently using differentiable rendering, and that they generalize to scenes with completely unseen geometry and illumination better than networks trained without appearance- matching regularization.