Neural Implicit Representations for Physical Parameter Inference from a Single Video

TL;DR

This paper introduces a novel method combining neural implicit representations and neural ODEs to infer physical parameters from a single video, enabling high-resolution analysis, interpretability, and realistic scene synthesis with minimal data.

Contribution

The work presents a single-video physical parameter inference method using neural implicit representations and neural ODEs, overcoming data requirements and enhancing interpretability and realism.

Findings

Physical parameters identified from one video

High-resolution, photo-realistic scene synthesis achieved

Long-term dynamic predictions demonstrated

Abstract

Neural networks have recently been used to analyze diverse physical systems and to identify the underlying dynamics. While existing methods achieve impressive results, they are limited by their strong demand for training data and their weak generalization abilities to out-of-distribution data. To overcome these limitations, in this work we propose to combine neural implicit representations for appearance modeling with neural ordinary differential equations (ODEs) for modelling physical phenomena to obtain a dynamic scene representation that can be identified directly from visual observations. Our proposed model combines several unique advantages: (i) Contrary to existing approaches that require large training datasets, we are able to identify physical parameters from only a single video. (ii) The use of neural implicit representations enables the processing of high-resolution videos and the synthesis of photo-realistic images. (iii) The embedded neural ODE has a known parametric form that allows for the identification of interpretable physical parameters, and (iv) long-term prediction in state space. (v) Furthermore, the photo-realistic rendering of novel scenes with modified physical parameters becomes possible.