NeX: Real-time View Synthesis with Neural Basis Expansion

TL;DR

NeX introduces a neural basis expansion technique for real-time, high-quality view synthesis that effectively models view-dependent effects and outperforms existing methods in speed and accuracy.

Contribution

The paper proposes a novel neural basis expansion approach for MPI that enhances view-dependent effect modeling and achieves real-time rendering with state-of-the-art quality.

Findings

Achieves the best scores on benchmark datasets.

Over 1000× faster rendering than previous methods.

Successfully models complex view-dependent effects like rainbow reflections.

Abstract

We present NeX, a new approach to novel view synthesis based on enhancements of multiplane image (MPI) that can reproduce next-level view-dependent effects -- in real time. Unlike traditional MPI that uses a set of simple RGB$\alpha$ planes, our technique models view-dependent effects by instead parameterizing each pixel as a linear combination of basis functions learned from a neural network. Moreover, we propose a hybrid implicit-explicit modeling strategy that improves upon fine detail and produces state-of-the-art results. Our method is evaluated on benchmark forward-facing datasets as well as our newly-introduced dataset designed to test the limit of view-dependent modeling with significantly more challenging effects such as rainbow reflections on a CD. Our method achieves the best overall scores across all major metrics on these datasets with more than 1000$\times$ faster rendering time than the state of the art. For real-time demos, visit https://nex-mpi.github.io/