Synthesizing Consistent Novel Views via 3D Epipolar Attention without Re-Training

TL;DR

This paper introduces a parameter-free method using epipolar geometry to improve the consistency of novel view synthesis from a single image, applicable without retraining and beneficial for 3D reconstruction.

Contribution

It proposes a novel, training-free approach leveraging epipolar attention for consistent view synthesis across multiple views, enhancing existing diffusion models.

Findings

Significantly improves view consistency in synthesis results.

Enhances downstream 3D reconstruction performance.

Operates without additional training or fine-tuning.

Abstract

Large diffusion models demonstrate remarkable zero-shot capabilities in novel view synthesis from a single image. However, these models often face challenges in maintaining consistency across novel and reference views. A crucial factor leading to this issue is the limited utilization of contextual information from reference views. Specifically, when there is an overlap in the viewing frustum between two views, it is essential to ensure that the corresponding regions maintain consistency in both geometry and appearance. This observation leads to a simple yet effective approach, where we propose to use epipolar geometry to locate and retrieve overlapping information from the input view. This information is then incorporated into the generation of target views, eliminating the need for training or fine-tuning, as the process requires no learnable parameters. Furthermore, to enhance the overall consistency of generated views, we extend the utilization of epipolar attention to a multi-view setting, allowing retrieval of overlapping information from the input view and other target views. Qualitative and quantitative experimental results demonstrate the effectiveness of our method in significantly improving the consistency of synthesized views without the need for any fine-tuning. Moreover, This enhancement also boosts the performance of downstream applications such as 3D reconstruction. The code is available at https://github.com/botaoye/ConsisSyn.