Rasterized Edge Gradients: Handling Discontinuities Differentiably

TL;DR

This paper introduces a novel differentiable rendering technique that accurately computes gradients at visibility discontinuities using micro-edges, improving the handling of complex geometries without altering the forward rendering process.

Contribution

The paper presents a new method for computing gradients at discontinuities in rasterization-based renderers using micro-edges, simplifying the process and maintaining image integrity.

Findings

Effective gradient computation at discontinuities in rasterization.

Applicable to masks, depth, and normals images without filtering.

Demonstrated success in dynamic human head scene reconstruction.

Abstract

Computing the gradients of a rendering process is paramount for diverse applications in computer vision and graphics. However, accurate computation of these gradients is challenging due to discontinuities and rendering approximations, particularly for surface-based representations and rasterization-based rendering. We present a novel method for computing gradients at visibility discontinuities for rasterization-based differentiable renderers. Our method elegantly simplifies the traditionally complex problem through a carefully designed approximation strategy, allowing for a straightforward, effective, and performant solution. We introduce a novel concept of micro-edges, which allows us to treat the rasterized images as outcomes of a differentiable, continuous process aligned with the inherently non-differentiable, discrete-pixel rasterization. This technique eliminates the necessity for rendering approximations or other modifications to the forward pass, preserving the integrity of the rendered image, which makes it applicable to rasterized masks, depth, and normals images where filtering is prohibitive. Utilizing micro-edges simplifies gradient interpretation at discontinuities and enables handling of geometry intersections, offering an advantage over the prior art. We showcase our method in dynamic human head scene reconstruction, demonstrating effective handling of camera images and segmentation masks.