Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes

TL;DR

This paper introduces Neural Scene Flow Fields, a novel neural representation for synthesizing novel views and times of dynamic scenes from monocular videos, capturing complex motions and effects.

Contribution

The paper proposes Neural Scene Flow Fields, a new continuous, neural-based representation for dynamic scenes that enables high-quality space-time view synthesis from monocular input.

Findings

Outperforms recent monocular view synthesis methods

Handles complex motions and thin structures effectively

Produces high-quality space-time visualizations

Abstract

We present a method to perform novel view and time synthesis of dynamic scenes, requiring only a monocular video with known camera poses as input. To do this, we introduce Neural Scene Flow Fields, a new representation that models the dynamic scene as a time-variant continuous function of appearance, geometry, and 3D scene motion. Our representation is optimized through a neural network to fit the observed input views. We show that our representation can be used for complex dynamic scenes, including thin structures, view-dependent effects, and natural degrees of motion. We conduct a number of experiments that demonstrate our approach significantly outperforms recent monocular view synthesis methods, and show qualitative results of space-time view synthesis on a variety of real-world videos.