End-to-end View Synthesis via NeRF Attention

TL;DR

This paper introduces NeRF Attention (NeRFA), a novel transformer-based approach for view synthesis that incorporates volumetric rendering principles and multi-stage attention to improve efficiency and quality over previous methods.

Contribution

NeRFA integrates NeRF-inspired feature modulation into transformers and employs multi-stage attention, achieving state-of-the-art results in view synthesis tasks.

Findings

NeRFA outperforms NeRF and NerFormer on four datasets.

NeRFA achieves new state-of-the-art in single-scene and category-centric view synthesis.

NeRFA effectively models volumetric rendering within a transformer framework.

Abstract

In this paper, we present a simple seq2seq formulation for view synthesis where we take a set of ray points as input and output colors corresponding to the rays. Directly applying a standard transformer on this seq2seq formulation has two limitations. First, the standard attention cannot successfully fit the volumetric rendering procedure, and therefore high-frequency components are missing in the synthesized views. Second, applying global attention to all rays and pixels is extremely inefficient. Inspired by the neural radiance field (NeRF), we propose the NeRF attention (NeRFA) to address the above problems. On the one hand, NeRFA considers the volumetric rendering equation as a soft feature modulation procedure. In this way, the feature modulation enhances the transformers with the NeRF-like inductive bias. On the other hand, NeRFA performs multi-stage attention to reduce the computational overhead. Furthermore, the NeRFA model adopts the ray and pixel transformers to learn the interactions between rays and pixels. NeRFA demonstrates superior performance over NeRF and NerFormer on four datasets: DeepVoxels, Blender, LLFF, and CO3D. Besides, NeRFA establishes a new state-of-the-art under two settings: the single-scene view synthesis and the category-centric novel view synthesis.