Learning Dynamic View Synthesis With Few RGBD Cameras

TL;DR

This paper introduces a novel method for dynamic view synthesis using RGBD cameras, employing feature point clouds, a neural renderer, and modules for depth inpainting and temporal consistency to produce high-quality free-viewpoint videos of indoor scenes.

Contribution

It presents a new RGBD-based approach with modules for depth inpainting and temporal stabilization, eliminating the need for prior models or scene-specific optimization.

Findings

Outperforms baseline in image fidelity

Achieves better spatial-temporal consistency

Demonstrates effectiveness on the HTI dataset

Abstract

There have been significant advancements in dynamic novel view synthesis in recent years. However, current deep learning models often require (1) prior models (e.g., SMPL human models), (2) heavy pre-processing, or (3) per-scene optimization. We propose to utilize RGBD cameras to remove these limitations and synthesize free-viewpoint videos of dynamic indoor scenes. We generate feature point clouds from RGBD frames and then render them into free-viewpoint videos via a neural renderer. However, the inaccurate, unstable, and incomplete depth measurements induce severe distortions, flickering, and ghosting artifacts. We enforce spatial-temporal consistency via the proposed Cycle Reconstruction Consistency and Temporal Stabilization module to reduce these artifacts. We introduce a simple Regional Depth-Inpainting module that adaptively inpaints missing depth values to render complete novel views. Additionally, we present a Human-Things Interactions dataset to validate our approach and facilitate future research. The dataset consists of 43 multi-view RGBD video sequences of everyday activities, capturing complex interactions between human subjects and their surroundings. Experiments on the HTI dataset show that our method outperforms the baseline per-frame image fidelity and spatial-temporal consistency. We will release our code, and the dataset on the website soon.