Differentiable Direct Volume Rendering

TL;DR

This paper introduces a fully differentiable volume rendering method that enables optimization of rendering parameters and volumetric data, facilitating automatic viewpoint selection and transfer function optimization with constant memory usage.

Contribution

It presents a novel differentiable volume renderer with constant memory footprint, enabling automatic optimization of rendering parameters and volumetric density fields.

Findings

Enables automatic viewpoint selection using differentiable entropy.

Optimizes volumetric density fields from images using absorption and emission models.

Achieves comparisons with algebraic reconstruction and differentiable path tracers.

Abstract

We present a differentiable volume rendering solution that provides differentiability of all continuous parameters of the volume rendering process. This differentiable renderer is used to steer the parameters towards a setting with an optimal solution of a problem-specific objective function. We have tailored the approach to volume rendering by enforcing a constant memory footprint via analytic inversion of the blending functions. This makes it independent of the number of sampling steps through the volume and facilitates the consideration of small-scale changes. The approach forms the basis for automatic optimizations regarding external parameters of the rendering process and the volumetric density field itself. We demonstrate its use for automatic viewpoint selection using differentiable entropy as objective, and for optimizing a transfer function from rendered images of a given volume. Optimization of per-voxel densities is addressed in two different ways: First, we mimic inverse tomography and optimize a 3D density field from images using an absorption model. This simplification enables comparisons with algebraic reconstruction techniques and state-of-the-art differentiable path tracers. Second, we introduce a novel approach for tomographic reconstruction from images using an emission-absorption model with post-shading via an arbitrary transfer function.