latentSplat: Autoencoding Variational Gaussians for Fast Generalizable 3D Reconstruction

TL;DR

latentSplat introduces a novel 3D reconstruction method that uses variational Gaussian representations in a latent space, enabling fast, scalable, and high-quality reconstruction from real video data.

Contribution

It combines regression and generative approaches with variational 3D Gaussians, allowing efficient encoding, sampling, and rendering for generalizable 3D reconstruction.

Findings

Outperforms previous methods in reconstruction quality

Achieves better generalization to large scenes and resolutions

Operates efficiently with high-resolution data

Abstract

We present latentSplat, a method to predict semantic Gaussians in a 3D latent space that can be splatted and decoded by a light-weight generative 2D architecture. Existing methods for generalizable 3D reconstruction either do not scale to large scenes and resolutions, or are limited to interpolation of close input views. latentSplat combines the strengths of regression-based and generative approaches while being trained purely on readily available real video data. The core of our method are variational 3D Gaussians, a representation that efficiently encodes varying uncertainty within a latent space consisting of 3D feature Gaussians. From these Gaussians, specific instances can be sampled and rendered via efficient splatting and a fast, generative decoder. We show that latentSplat outperforms previous works in reconstruction quality and generalization, while being fast and scalable to high-resolution data.