GaussianVAE: Adaptive Learning Dynamics of 3D Gaussians for High-Fidelity Super-Resolution

TL;DR

This paper introduces GaussianVAE, a lightweight generative model that enhances 3D Gaussian Splatting resolution and detail beyond training limits, using Hessian-assisted sampling for efficient, real-time super-resolution.

Contribution

The paper proposes a novel Hessian-assisted sampling strategy within a generative model to improve 3D Gaussian Splatting resolution and detail in real-time, surpassing previous limitations.

Findings

Significant improvements in geometric accuracy and rendering quality.

Real-time performance at 0.015s per inference on a consumer GPU.

Outperforms state-of-the-art methods in super-resolution of 3D scenes.

Abstract

We present a novel approach for enhancing the resolution and geometric fidelity of 3D Gaussian Splatting (3DGS) beyond native training resolution. Current 3DGS methods are fundamentally limited by their input resolution, producing reconstructions that cannot extrapolate finer details than are present in the training views. Our work breaks this limitation through a lightweight generative model that predicts and refines additional 3D Gaussians where needed most. The key innovation is our Hessian-assisted sampling strategy, which intelligently identifies regions that are likely to benefit from densification, ensuring computational efficiency. Unlike computationally intensive GANs or diffusion approaches, our method operates in real-time (0.015s per inference on a single consumer-grade GPU), making it practical for interactive applications. Comprehensive experiments demonstrate significant improvements in both geometric accuracy and rendering quality compared to state-of-the-art methods, establishing a new paradigm for resolution-free 3D scene enhancement.