From NeRFs to Gaussian Splats, and Back

TL;DR

This paper introduces a method to convert between neural radiance fields (NeRFs) and Gaussian splatting (GS), combining NeRFs' view generalization with GS's fast rendering for improved robotics applications.

Contribution

It presents a novel procedure for bidirectional conversion between NeRFs and GS, enabling the use of both methods' advantages in a single framework.

Findings

Achieves superior view synthesis quality on dissimilar views.

Enables real-time rendering with Gaussian splats.

Maintains low computational cost for conversions.

Abstract

For robotics applications where there is a limited number of (typically ego-centric) views, parametric representations such as neural radiance fields (NeRFs) generalize better than non-parametric ones such as Gaussian splatting (GS) to views that are very different from those in the training data; GS however can render much faster than NeRFs. We develop a procedure to convert back and forth between the two. Our approach achieves the best of both NeRFs (superior PSNR, SSIM, and LPIPS on dissimilar views, and a compact representation) and GS (real-time rendering and ability for easily modifying the representation); the computational cost of these conversions is minor compared to training the two from scratch.