Physics-informed Guided Disentanglement in Generative Networks

TL;DR

This paper introduces a physics-informed framework for disentangling visual traits in image translation networks, improving controllability and quality by leveraging physical models or neural guidance.

Contribution

It presents a novel, versatile approach to disentanglement using physics models or neural networks, enhancing image translation performance in challenging scenarios.

Findings

Significant qualitative improvements in image translation quality.

Quantitative performance gains demonstrated across multiple scenarios.

Effective disentanglement achieved with physics-guided and neural-guided strategies.

Abstract

Image-to-image translation (i2i) networks suffer from entanglement effects in presence of physics-related phenomena in target domain (such as occlusions, fog, etc), lowering altogether the translation quality, controllability and variability. In this paper, we propose a general framework to disentangle visual traits in target images. Primarily, we build upon collection of simple physics models, guiding the disentanglement with a physical model that renders some of the target traits, and learning the remaining ones. Because physics allows explicit and interpretable outputs, our physical models (optimally regressed on target) allows generating unseen scenarios in a controllable manner. Secondarily, we show the versatility of our framework to neural-guided disentanglement where a generative network is used in place of a physical model in case the latter is not directly accessible. Altogether, we introduce three strategies of disentanglement being guided from either a fully differentiable physics model, a (partially) non-differentiable physics model, or a neural network. The results show our disentanglement strategies dramatically increase performances qualitatively and quantitatively in several challenging scenarios for image translation.