Inferring Fluid Dynamics via Inverse Rendering

TL;DR

This paper presents a novel method for inferring fluid dynamics from videos using a differentiable simulation and rendering framework trained without ground-truth fluid data, enabling inverse rendering of fluid scenes.

Contribution

It introduces a differentiable Euler simulator integrated with a volumetric renderer, allowing end-to-end inference of fluid dynamics from unannotated videos without supervision.

Findings

Effective inference of fluid dynamics from videos.

Good generalization demonstrated on multiple datasets.

End-to-end differentiable framework for fluid inverse rendering.

Abstract

Humans have a strong intuitive understanding of physical processes such as fluid falling by just a glimpse of such a scene picture, i.e., quickly derived from our immersive visual experiences in memory. This work achieves such a photo-to-fluid-dynamics reconstruction functionality learned from unannotated videos, without any supervision of ground-truth fluid dynamics. In a nutshell, a differentiable Euler simulator modeled with a ConvNet-based pressure projection solver, is integrated with a volumetric renderer, supporting end-to-end/coherent differentiable dynamic simulation and rendering. By endowing each sampled point with a fluid volume value, we derive a NeRF-like differentiable renderer dedicated from fluid data; and thanks to this volume-augmented representation, fluid dynamics could be inversely inferred from the error signal between the rendered result and ground-truth video frame (i.e., inverse rendering). Experiments on our generated Fluid Fall datasets and DPI Dam Break dataset are conducted to demonstrate both effectiveness and generalization ability of our method.