GoLF-NRT: Integrating Global Context and Local Geometry for Few-Shot View Synthesis

TL;DR

GoLF-NRT introduces a novel neural rendering transformer that combines global scene context and local geometric features to significantly improve few-shot view synthesis quality, even with as few as one to three input views.

Contribution

This paper presents a new global-local feature fusion approach with an adaptive sampling strategy for neural rendering, enhancing performance in few-shot view synthesis scenarios.

Findings

Achieves state-of-the-art results on public datasets

Effective with as few as 1-3 input views

Outperforms existing generalizable NeRF models

Abstract

Neural Radiance Fields (NeRF) have transformed novel view synthesis by modeling scene-specific volumetric representations directly from images. While generalizable NeRF models can generate novel views across unknown scenes by learning latent ray representations, their performance heavily depends on a large number of multi-view observations. However, with limited input views, these methods experience significant degradation in rendering quality. To address this limitation, we propose GoLF-NRT: a Global and Local feature Fusion-based Neural Rendering Transformer. GoLF-NRT enhances generalizable neural rendering from few input views by leveraging a 3D transformer with efficient sparse attention to capture global scene context. In parallel, it integrates local geometric features extracted along the epipolar line, enabling high-quality scene reconstruction from as few as 1 to 3 input views. Furthermore, we introduce an adaptive sampling strategy based on attention weights and kernel regression, improving the accuracy of transformer-based neural rendering. Extensive experiments on public datasets show that GoLF-NRT achieves state-of-the-art performance across varying numbers of input views, highlighting the effectiveness and superiority of our approach. Code is available at https://github.com/KLMAV-CUC/GoLF-NRT.