ViewFormer: NeRF-free Neural Rendering from Few Images Using Transformers

TL;DR

ViewFormer introduces a 2D transformer-based approach for novel view synthesis from few images, avoiding 3D reasoning, reducing training time, and enabling camera pose estimation within a unified model.

Contribution

The paper presents a NeRF-free, 2D-only neural rendering method using transformers and a novel branching attention mechanism for efficient training and pose estimation.

Findings

Competitive results with NeRF-based methods on real-world scenes

Faster training times due to 2D-only approach

Unified model for view synthesis and camera pose estimation

Abstract

Novel view synthesis is a long-standing problem. In this work, we consider a variant of the problem where we are given only a few context views sparsely covering a scene or an object. The goal is to predict novel viewpoints in the scene, which requires learning priors. The current state of the art is based on Neural Radiance Field (NeRF), and while achieving impressive results, the methods suffer from long training times as they require evaluating millions of 3D point samples via a neural network for each image. We propose a 2D-only method that maps multiple context views and a query pose to a new image in a single pass of a neural network. Our model uses a two-stage architecture consisting of a codebook and a transformer model. The codebook is used to embed individual images into a smaller latent space, and the transformer solves the view synthesis task in this more compact space. To train our model efficiently, we introduce a novel branching attention mechanism that allows us to use the same model not only for neural rendering but also for camera pose estimation. Experimental results on real-world scenes show that our approach is competitive compared to NeRF-based methods while not reasoning explicitly in 3D, and it is faster to train.