LiftRefine: Progressively Refined View Synthesis from 3D Lifting with Volume-Triplane Representations

TL;DR

LiftRefine introduces a progressive view synthesis approach combining 3D lifting, tri-plane representations, and diffusion models to generate high-quality novel views from limited input images.

Contribution

The paper presents a novel two-stage method with a progressive refinement technique that enhances 3D view synthesis from few-view images using volume-triplane representations.

Findings

Outperforms state-of-the-art on multiple datasets

Achieves high-quality, multi-view consistent renderings

Efficiently synthesizes novel views from limited inputs

Abstract

We propose a new view synthesis method via synthesizing a 3D neural field from both single or few-view input images. To address the ill-posed nature of the image-to-3D generation problem, we devise a two-stage method that involves a reconstruction model and a diffusion model for view synthesis. Our reconstruction model first lifts one or more input images to the 3D space from a volume as the coarse-scale 3D representation followed by a tri-plane as the fine-scale 3D representation. To mitigate the ambiguity in occluded regions, our diffusion model then hallucinates missing details in the rendered images from tri-planes. We then introduce a new progressive refinement technique that iteratively applies the reconstruction and diffusion model to gradually synthesize novel views, boosting the overall quality of the 3D representations and their rendering. Empirical evaluation demonstrates the superiority of our method over state-of-the-art methods on the synthetic SRN-Car dataset, the in-the-wild CO3D dataset, and large-scale Objaverse dataset while achieving both sampling efficacy and multi-view consistency.