Dynamic View Synthesis from Small Camera Motion Videos

TL;DR

This paper introduces a novel approach for dynamic view synthesis from videos with limited camera motion, employing distribution-based depth regularization and camera parameter learning to improve scene geometry and rendering accuracy.

Contribution

It proposes a new Distribution-based Depth Regularization (DDR) and camera parameter learning to handle small camera motion scenarios in view synthesis.

Findings

Outperforms state-of-the-art methods on small camera motion datasets

Effectively learns scene geometry with limited camera movement

Improves robustness of camera parameter estimation during training

Abstract

Novel view synthesis for dynamic $3$D scenes poses a significant challenge. Many notable efforts use NeRF-based approaches to address this task and yield impressive results. However, these methods rely heavily on sufficient motion parallax in the input images or videos. When the camera motion range becomes limited or even stationary (i.e., small camera motion), existing methods encounter two primary challenges: incorrect representation of scene geometry and inaccurate estimation of camera parameters. These challenges make prior methods struggle to produce satisfactory results or even become invalid. To address the first challenge, we propose a novel Distribution-based Depth Regularization (DDR) that ensures the rendering weight distribution to align with the true distribution. Specifically, unlike previous methods that use depth loss to calculate the error of the expectation, we calculate the expectation of the error by using Gumbel-softmax to differentiably sample points from discrete rendering weight distribution. Additionally, we introduce constraints that enforce the volume density of spatial points before the object boundary along the ray to be near zero, ensuring that our model learns the correct geometry of the scene. To demystify the DDR, we further propose a visualization tool that enables observing the scene geometry representation at the rendering weight level. For the second challenge, we incorporate camera parameter learning during training to enhance the robustness of our model to camera parameters. We conduct extensive experiments to demonstrate the effectiveness of our approach in representing scenes with small camera motion input, and our results compare favorably to state-of-the-art methods.