IBRNet: Learning Multi-View Image-Based Rendering

TL;DR

IBRNet is a neural network architecture that synthesizes novel views of complex scenes by interpolating sparse source views, combining classic image-based rendering principles with modern neural techniques for high-quality, generalizable view synthesis.

Contribution

We introduce a neural network with a ray transformer that estimates radiance and density from multiple views, enabling generalizable, high-resolution novel view synthesis from sparse inputs.

Findings

Outperforms recent generalizable view synthesis methods.

Competitive with single-scene neural rendering when fine-tuned.

Effective volume rendering trained with multi-view posed images.

Abstract

We present a method that synthesizes novel views of complex scenes by interpolating a sparse set of nearby views. The core of our method is a network architecture that includes a multilayer perceptron and a ray transformer that estimates radiance and volume density at continuous 5D locations (3D spatial locations and 2D viewing directions), drawing appearance information on the fly from multiple source views. By drawing on source views at render time, our method hearkens back to classic work on image-based rendering (IBR), and allows us to render high-resolution imagery. Unlike neural scene representation work that optimizes per-scene functions for rendering, we learn a generic view interpolation function that generalizes to novel scenes. We render images using classic volume rendering, which is fully differentiable and allows us to train using only multi-view posed images as supervision. Experiments show that our method outperforms recent novel view synthesis methods that also seek to generalize to novel scenes. Further, if fine-tuned on each scene, our method is competitive with state-of-the-art single-scene neural rendering methods. Project page: https://ibrnet.github.io/