MIMO-NeRF: Fast Neural Rendering with Multi-input Multi-output Neural Radiance Fields

TL;DR

MIMO-NeRF introduces a multi-input multi-output neural radiance field model that accelerates rendering by reducing the number of neural networks needed, using a self-supervised approach to maintain quality.

Contribution

It replaces traditional SISO MLPs with MIMO MLPs for faster rendering and proposes a self-supervised regularization method to address ambiguity issues.

Findings

Achieves a good speed-quality trade-off in rendering.

Compatible with existing fast NeRF methods like DONeRF and TensoRF.

Demonstrates efficiency with reasonable training time.

Abstract

Neural radiance fields (NeRFs) have shown impressive results for novel view synthesis. However, they depend on the repetitive use of a single-input single-output multilayer perceptron (SISO MLP) that maps 3D coordinates and view direction to the color and volume density in a sample-wise manner, which slows the rendering. We propose a multi-input multi-output NeRF (MIMO-NeRF) that reduces the number of MLPs running by replacing the SISO MLP with a MIMO MLP and conducting mappings in a group-wise manner. One notable challenge with this approach is that the color and volume density of each point can differ according to a choice of input coordinates in a group, which can lead to some notable ambiguity. We also propose a self-supervised learning method that regularizes the MIMO MLP with multiple fast reformulated MLPs to alleviate this ambiguity without using pretrained models. The results of a comprehensive experimental evaluation including comparative and ablation studies are presented to show that MIMO-NeRF obtains a good trade-off between speed and quality with a reasonable training time. We then demonstrate that MIMO-NeRF is compatible with and complementary to previous advancements in NeRFs by applying it to two representative fast NeRFs, i.e., a NeRF with sample reduction (DONeRF) and a NeRF with alternative representations (TensoRF).