Zero Grads: Learning Local Surrogate Losses for Non-Differentiable Graphics

TL;DR

ZeroGrads introduces a neural surrogate framework that enables gradient-based optimization of non-differentiable graphics problems by learning local approximations, improving scalability and efficiency in complex black-box scenarios.

Contribution

The paper presents ZeroGrads, a novel method that automatically learns local surrogate losses for non-differentiable graphics tasks, allowing efficient gradient-based optimization without pre-training.

Findings

Successfully optimized non-differentiable graphics problems including rendering and physics.

Scales to high-dimensional problems with up to 35,000 variables.

Achieves competitive performance with efficient sampling schemes.

Abstract

Gradient-based optimization is now ubiquitous across graphics, but unfortunately can not be applied to problems with undefined or zero gradients. To circumvent this issue, the loss function can be manually replaced by a ``surrogate'' that has similar minima but is differentiable. Our proposed framework, ZeroGrads, automates this process by learning a neural approximation of the objective function, which in turn can be used to differentiate through arbitrary black-box graphics pipelines. We train the surrogate on an actively smoothed version of the objective and encourage locality, focusing the surrogate's capacity on what matters at the current training episode. The fitting is performed online, alongside the parameter optimization, and self-supervised, without pre-computed data or pre-trained models. As sampling the objective is expensive (it requires a full rendering or simulator run), we devise an efficient sampling scheme that allows for tractable run-times and competitive performance at little overhead. We demonstrate optimizing diverse non-convex, non-differentiable black-box problems in graphics, such as visibility in rendering, discrete parameter spaces in procedural modelling or optimal control in physics-driven animation. In contrast to other derivative-free algorithms, our approach scales well to higher dimensions, which we demonstrate on problems with up to 35k interlinked variables.