Geometry-guided Online 3D Video Synthesis with Multi-View Temporal Consistency

TL;DR

This paper presents a geometry-guided online 3D video synthesis method that ensures view and temporal consistency, improving quality and efficiency over traditional dense multi-view approaches by leveraging global geometry and depth refinement.

Contribution

It introduces a novel geometry-guided rendering pipeline that refines depth maps and guides a blending network for consistent, high-quality online 3D video synthesis.

Findings

Achieves high-quality, view, and temporal consistency in video synthesis.

Runs efficiently in an online setting with reduced computational cost.

Outperforms traditional dense multi-view methods in quality and speed.

Abstract

We introduce a novel geometry-guided online video view synthesis method with enhanced view and temporal consistency. Traditional approaches achieve high-quality synthesis from dense multi-view camera setups but require significant computational resources. In contrast, selective-input methods reduce this cost but often compromise quality, leading to multi-view and temporal inconsistencies such as flickering artifacts. Our method addresses this challenge to deliver efficient, high-quality novel-view synthesis with view and temporal consistency. The key innovation of our approach lies in using global geometry to guide an image-based rendering pipeline. To accomplish this, we progressively refine depth maps using color difference masks across time. These depth maps are then accumulated through truncated signed distance fields in the synthesized view's image space. This depth representation is view and temporally consistent, and is used to guide a pre-trained blending network that fuses multiple forward-rendered input-view images. Thus, the network is encouraged to output geometrically consistent synthesis results across multiple views and time. Our approach achieves consistent, high-quality video synthesis, while running efficiently in an online manner.