Learn2Splat: Extending the Horizon of Learned 3DGS Optimization

TL;DR

Learn2Splat introduces a novel meta-learning based learned optimizer for 3D Gaussian Splatting that enhances convergence speed and stability over long optimization horizons, improving 3D scene reconstruction quality.

Contribution

It presents the first learned optimizer for 3DGS that maintains stability over extended iterations without auxiliary mechanisms, using a new meta-learning scheme and architecture.

Findings

Improved early novel view synthesis quality.

Stable optimization over long horizons.

Zero-shot generalization to unseen settings.

Abstract

3D Gaussian Splatting (3DGS) optimization is most commonly performed using standard optimizers (Adam, SGD). While stable across diverse scenes, standard optimizers are general-purpose and not tailored to the structure of the problem. In particular, they produce independent parameter updates that do not capture the structural and spatial relationships within a scene, leading to inefficient optimization and slow convergence. Recent works introduced learned optimizers that predict correlated updates informed by inter-parameter and inter-Gaussian dependencies. However, these methods are trained for a fixed number of optimization iterations and rely on manually scheduled learning rates to avoid degradation. In this paper, we introduce a learned optimizer for 3DGS that avoids degradation over extended optimization horizons without auxiliary mechanisms. To enable this, we propose a meta-learning scheme that extends the optimization horizon via a checkpoint buffer and an optimizer rollout strategy, combined with an architecture that encodes gradient scale information in its latent states. Results show improved early novel view synthesis quality while remaining stable over long horizons, with zero-shot generalization to unseen reconstruction settings. To support our findings, we introduce the first unified framework for training and evaluating both learned and conventional optimizers across sparse and dense view settings. Code and models will be released publicly. Our project page is available at https://naamapearl.github.io/learn2splat .