Neural Scene Graphs for Dynamic Scenes

TL;DR

This paper introduces a neural rendering approach that decomposes dynamic scenes into scene graphs, enabling realistic view synthesis and object manipulation from video data, extending static scene methods to dynamic, object-based representations.

Contribution

It presents the first neural rendering method that models dynamic scenes with scene graphs, encoding object transformations and radiance for flexible scene manipulation.

Findings

Successfully models dynamic scenes from video data

Enables rendering of novel views with unseen objects and poses

Validates on synthetic and real automotive datasets

Abstract

Recent implicit neural rendering methods have demonstrated that it is possible to learn accurate view synthesis for complex scenes by predicting their volumetric density and color supervised solely by a set of RGB images. However, existing methods are restricted to learning efficient representations of static scenes that encode all scene objects into a single neural network, and lack the ability to represent dynamic scenes and decompositions into individual scene objects. In this work, we present the first neural rendering method that decomposes dynamic scenes into scene graphs. We propose a learned scene graph representation, which encodes object transformation and radiance, to efficiently render novel arrangements and views of the scene. To this end, we learn implicitly encoded scenes, combined with a jointly learned latent representation to describe objects with a single implicit function. We assess the proposed method on synthetic and real automotive data, validating that our approach learns dynamic scenes -- only by observing a video of this scene -- and allows for rendering novel photo-realistic views of novel scene compositions with unseen sets of objects at unseen poses.