Projected Representation Conditioning for High-fidelity Novel View Synthesis

TL;DR

This paper introduces ReNoV, a diffusion-based framework that uses external visual representations to improve the geometric consistency and quality of novel view synthesis, especially from sparse image collections.

Contribution

The paper presents a novel representation-guided approach with dedicated projection modules that enhance diffusion-based view synthesis by leveraging external representations.

Findings

Improved reconstruction fidelity and inpainting quality.

Outperforms prior diffusion-based methods on benchmarks.

Enables robust synthesis from sparse, unposed images.

Abstract

We propose a novel framework for diffusion-based novel view synthesis in which we leverage external representations as conditions, harnessing their geometric and semantic correspondence properties for enhanced geometric consistency in generated novel viewpoints. First, we provide a detailed analysis exploring the correspondence capabilities emergent in the spatial attention of external visual representations. Building from these insights, we propose a representation-guided novel view synthesis through dedicated representation projection modules that inject external representations into the diffusion process, a methodology named ReNoV, short for representation-guided novel view synthesis. Our experiments show that this design yields marked improvements in both reconstruction fidelity and inpainting quality, outperforming prior diffusion-based novel-view methods on standard benchmarks and enabling robust synthesis from sparse, unposed image collections.