ID-NeRF: Indirect Diffusion-guided Neural Radiance Fields for Generalizable View Synthesis

TL;DR

ID-NeRF introduces a diffusion-guided approach that leverages pre-trained priors and an indirect injection strategy to enhance generalizable novel view synthesis from sparse inputs, overcoming limitations of previous NeRF methods.

Contribution

The paper proposes ID-NeRF, a novel framework that uses pre-trained diffusion priors and an indirect prior injection strategy to improve 3D-consistent view synthesis from sparse data.

Findings

Outperforms previous methods in sparse view synthesis scenarios.

Effectively incorporates pre-trained priors for high-quality results.

Demonstrates robustness across multiple datasets.

Abstract

Implicit neural representations, represented by Neural Radiance Fields (NeRF), have dominated research in 3D computer vision by virtue of high-quality visual results and data-driven benefits. However, their realistic applications are hindered by the need for dense inputs and per-scene optimization. To solve this problem, previous methods implement generalizable NeRFs by extracting local features from sparse inputs as conditions for the NeRF decoder. However, although this way can allow feed-forward reconstruction, they suffer from the inherent drawback of yielding sub-optimal results caused by erroneous reprojected features. In this paper, we focus on this problem and aim to address it by introducing pre-trained generative priors to enable high-quality generalizable novel view synthesis. Specifically, we propose a novel Indirect Diffusion-guided NeRF framework, termed ID-NeRF, which leverages pre-trained diffusion priors as a guide for the reprojected features created by the previous paradigm. Notably, to enable 3D-consistent predictions, the proposed ID-NeRF discards the way of direct supervision commonly used in prior 3D generative models and instead adopts a novel indirect prior injection strategy. This strategy is implemented by distilling pre-trained knowledge into an imaginative latent space via score-based distillation, and an attention-based refinement module is then proposed to leverage the embedded priors to improve reprojected features extracted from sparse inputs. We conduct extensive experiments on multiple datasets to evaluate our method, and the results demonstrate the effectiveness of our method in synthesizing novel views in a generalizable manner, especially in sparse settings.