NeRF Is a Valuable Assistant for 3D Gaussian Splatting

TL;DR

This paper presents NeRF-GS, a hybrid framework combining Neural Radiance Fields and 3D Gaussian Splatting to improve 3D scene representation, surpassing existing methods with state-of-the-art results.

Contribution

NeRF-GS is the first framework to jointly optimize NeRF and 3D Gaussian Splatting, addressing their limitations and leveraging their complementarity for enhanced 3D scene modeling.

Findings

NeRF-GS outperforms existing methods on benchmark datasets.

The hybrid approach achieves state-of-the-art performance.

NeRF and 3DGS are shown to be complementary techniques.

Abstract

We introduce NeRF-GS, a novel framework that jointly optimizes Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS). This framework leverages the inherent continuous spatial representation of NeRF to mitigate several limitations of 3DGS, including sensitivity to Gaussian initialization, limited spatial awareness, and weak inter-Gaussian correlations, thereby enhancing its performance. In NeRF-GS, we revisit the design of 3DGS and progressively align its spatial features with NeRF, enabling both representations to be optimized within the same scene through shared 3D spatial information. We further address the formal distinctions between the two approaches by optimizing residual vectors for both implicit features and Gaussian positions to enhance the personalized capabilities of 3DGS. Experimental results on benchmark datasets show that NeRF-GS surpasses existing methods and achieves state-of-the-art performance. This outcome confirms that NeRF and 3DGS are complementary rather than competing, offering new insights into hybrid approaches that combine 3DGS and NeRF for efficient 3D scene representation.