Differentiable Rendering: A Survey

TL;DR

This survey reviews differentiable rendering, a technique enabling gradient-based optimization of 3D objects from images, reducing data annotation needs and advancing applications in vision tasks.

Contribution

It provides a comprehensive overview of differentiable rendering, highlighting its current state, applications, and open research challenges.

Findings

Differentiable rendering enables gradient-based 3D object optimization.

It reduces the need for extensive 3D data collection and annotation.

The survey identifies key open problems and future directions.

Abstract

Deep neural networks (DNNs) have shown remarkable performance improvements on vision-related tasks such as object detection or image segmentation. Despite their success, they generally lack the understanding of 3D objects which form the image, as it is not always possible to collect 3D information about the scene or to easily annotate it. Differentiable rendering is a novel field which allows the gradients of 3D objects to be calculated and propagated through images. It also reduces the requirement of 3D data collection and annotation, while enabling higher success rate in various applications. This paper reviews existing literature and discusses the current state of differentiable rendering, its applications and open research problems.