Diff3R: Feed-forward 3D Gaussian Splatting with Uncertainty-aware Differentiable Optimization

TL;DR

Diff3R introduces a novel framework that combines fast feed-forward 3D Gaussian Splatting with test-time optimization, using differentiable layers and uncertainty modeling to enhance rendering quality and robustness.

Contribution

The paper presents a differentiable optimization layer integrated into 3D Gaussian Splatting, enabling improved initialization and robustness for test-time optimization.

Findings

Enhanced rendering quality with the integrated optimization layer.

Robustness against input outliers improved by uncertainty modeling.

Seamless integration into existing architectures yields consistent performance gains.

Abstract

Recent advances in 3D Gaussian Splatting (3DGS) present two main directions: feed-forward models offer fast inference in sparse-view settings, while per-scene optimization yields high-quality renderings but is computationally expensive. To combine the benefits of both, we introduce Diff3R, a novel framework that explicitly bridges feed-forward prediction and test-time optimization. By incorporating a differentiable 3DGS optimization layer directly into the training loop, our network learns to predict an optimal initialization for test-time optimization rather than a conventional zero-shot result. To overcome the computational cost of backpropagating through the optimization steps, we propose computing gradients via the Implicit Function Theorem and a scalable, matrix-free PCG solver tailored for 3DGS optimization. Additionally, we incorporate a data-driven uncertainty model into the optimization process by adaptively controlling how much the parameters are allowed to change during optimization. This approach effectively mitigates overfitting in under-constrained regions and increases robustness against input outliers. Since our proposed optimization layer is model-agnostic, we show that it can be seamlessly integrated into existing feed-forward 3DGS architectures for both pose-given and pose-free methods, providing improvements for test-time optimization.