3DGS$^2$: Near Second-order Converging 3D Gaussian Splatting

TL;DR

This paper introduces a near second-order converging training algorithm for 3D Gaussian Splatting, significantly accelerating training speed while maintaining high-quality 3D scene reconstruction.

Contribution

It develops a novel optimization method exploiting Gaussian kernel properties to achieve faster convergence in 3DGS training without global Hessian computation.

Findings

Achieves over 10x fewer training iterations compared to standard SGD-based 3DGS.

Maintains or improves reconstruction quality with faster convergence.

Utilizes local Newton systems and spatial structure to enhance training efficiency.

Abstract

3D Gaussian Splatting (3DGS) has emerged as a mainstream solution for novel view synthesis and 3D reconstruction. By explicitly encoding a 3D scene using a collection of Gaussian kernels, 3DGS achieves high-quality rendering with superior efficiency. As a learning-based approach, 3DGS training has been dealt with the standard stochastic gradient descent (SGD) method, which offers at most linear convergence. Consequently, training often requires tens of minutes, even with GPU acceleration. This paper introduces a (near) second-order convergent training algorithm for 3DGS, leveraging its unique properties. Our approach is inspired by two key observations. First, the attributes of a Gaussian kernel contribute independently to the image-space loss, which endorses isolated and local optimization algorithms. We exploit this by splitting the optimization at the level of individual kernel attributes, analytically constructing small-size Newton systems for each parameter group, and efficiently solving these systems on GPU threads. This achieves Newton-like convergence per training image without relying on the global Hessian. Second, kernels exhibit sparse and structured coupling across input images. This property allows us to effectively utilize spatial information to mitigate overshoot during stochastic training. Our method converges an order faster than standard GPU-based 3DGS training, requiring over $10\times$ fewer iterations while maintaining or surpassing the quality of the compared with the SGD-based 3DGS reconstructions.