Toy-GS: Assembling Local Gaussians for Precisely Rendering Large-Scale Free Camera Trajectories

TL;DR

Toy-GS introduces a novel adaptive spatial division and local Gaussian approach for precise, memory-efficient rendering of large-scale free camera trajectories, achieving state-of-the-art results.

Contribution

The paper proposes an adaptive spatial division method and regional fusion of local and global Gaussians for improved large-scale scene rendering.

Findings

Achieves 1.19 dB PSNR improvement over benchmarks.

Reduces GPU memory usage by 7 GB.

Effective on multiple large-scale datasets.

Abstract

Currently, 3D rendering for large-scale free camera trajectories, namely, arbitrary input camera trajectories, poses significant challenges: 1) The distribution and observation angles of the cameras are irregular, and various types of scenes are included in the free trajectories; 2) Processing the entire point cloud and all images at once for large-scale scenes requires a substantial amount of GPU memory. This paper presents a Toy-GS method for accurately rendering large-scale free camera trajectories. Specifically, we propose an adaptive spatial division approach for free trajectories to divide cameras and the sparse point cloud of the entire scene into various regions according to camera poses. Training each local Gaussian in parallel for each area enables us to concentrate on texture details and minimize GPU memory usage. Next, we use the multi-view constraint and position-aware point adaptive control (PPAC) to improve the rendering quality of texture details. In addition, our regional fusion approach combines local and global Gaussians to enhance rendering quality with an increasing number of divided areas. Extensive experiments have been carried out to confirm the effectiveness and efficiency of Toy-GS, leading to state-of-the-art results on two public large-scale datasets as well as our SCUTic dataset. Our proposal demonstrates an enhancement of 1.19 dB in PSNR and conserves 7 G of GPU memory when compared to various benchmarks.