PyGS: Large-scale Scene Representation with Pyramidal 3D Gaussian Splatting

TL;DR

PyGS introduces a hierarchical pyramidal Gaussian representation initialized with NeRF, enabling efficient large-scale scene rendering with high detail and over 400 times faster performance than existing methods.

Contribution

The paper proposes Pyramidal 3D Gaussian Splatting (PyGS) with NeRF initialization, addressing scalability and efficiency challenges in large-scale scene representation.

Findings

Achieves over 400x faster rendering than state-of-the-art methods.

Effectively captures multi-scale scene details with pyramidal Gaussian structure.

Demonstrates significant performance improvements across multiple large-scale datasets.

Abstract

Neural Radiance Fields (NeRFs) have demonstrated remarkable proficiency in synthesizing photorealistic images of large-scale scenes. However, they are often plagued by a loss of fine details and long rendering durations. 3D Gaussian Splatting has recently been introduced as a potent alternative, achieving both high-fidelity visual results and accelerated rendering performance. Nonetheless, scaling 3D Gaussian Splatting is fraught with challenges. Specifically, large-scale scenes grapples with the integration of objects across multiple scales and disparate viewpoints, which often leads to compromised efficacy as the Gaussians need to balance between detail levels. Furthermore, the generation of initialization points via COLMAP from large-scale dataset is both computationally demanding and prone to incomplete reconstructions. To address these challenges, we present Pyramidal 3D Gaussian Splatting (PyGS) with NeRF Initialization. Our approach represent the scene with a hierarchical assembly of Gaussians arranged in a pyramidal fashion. The top level of the pyramid is composed of a few large Gaussians, while each subsequent layer accommodates a denser collection of smaller Gaussians. We effectively initialize these pyramidal Gaussians through sampling a rapidly trained grid-based NeRF at various frequencies. We group these pyramidal Gaussians into clusters and use a compact weighting network to dynamically determine the influence of each pyramid level of each cluster considering camera viewpoint during rendering. Our method achieves a significant performance leap across multiple large-scale datasets and attains a rendering time that is over 400 times faster than current state-of-the-art approaches.