Complete Gaussian Splats from a Single Image with Denoising Diffusion Models

TL;DR

This paper introduces a generative approach using diffusion models to reconstruct complete 3D scenes from a single image, effectively handling occlusions and unobserved areas with diverse, high-quality outputs.

Contribution

It presents a novel generative framework with a Variational AutoReconstructor and diffusion model to produce diverse 3D Gaussian splat reconstructions from a single image, overcoming data limitations.

Findings

Produces faithful 3D reconstructions from a single image.

Generates diverse plausible surface completions.

Enables high-quality 360-degree scene renderings.

Abstract

Gaussian splatting typically requires dense observations of the scene and can fail to reconstruct occluded and unobserved areas. We propose a latent diffusion model to reconstruct a complete 3D scene with Gaussian splats, including the occluded parts, from only a single image during inference. Completing the unobserved surfaces of a scene is challenging due to the ambiguity of the plausible surfaces. Conventional methods use a regression-based formulation to predict a single "mode" for occluded and out-of-frustum surfaces, leading to blurriness, implausibility, and failure to capture multiple possible explanations. Thus, they often address this problem partially, focusing either on objects isolated from the background, reconstructing only visible surfaces, or failing to extrapolate far from the input views. In contrast, we propose a generative formulation to learn a distribution of 3D representations of Gaussian splats conditioned on a single input image. To address the lack of ground-truth training data, we propose a Variational AutoReconstructor to learn a latent space only from 2D images in a self-supervised manner, over which a diffusion model is trained. Our method generates faithful reconstructions and diverse samples with the ability to complete the occluded surfaces for high-quality 360-degree renderings.

Tables

Table 1. Table 1 : Comparison to Related Works. The table highlights the main differences from closely related baselines. We propose a generative method with diffusion models to reconstruct 3D scenes with Gaussian splats in real time from a single image.

Scale	Methods	Outputs	Inputs	Formulations	Architectures	3D Representations	Computation
Objects	Objects Models [73, 79, 2, 84]	3D	/	/	/	GS, NeRF	/
	Novel View Sythesis [20, 6, 27]	Image	/	/	/		/
	Video Generation [44, 37]	Video	/	/	/	None	/
	SplatterImage [61, 60, 7, 8]	3D	1 or 2-views	Regression-based	/	GS	Real-time
	LatentSplat [70]		2-views		VAE	GS	Real-time
	CAT3D [19]		1-view		2D Diffusion	GS	Offline
	DFM [65]		1-view		3D Diffusion	NeRF	Offline
Scenes	Ours	3D	1-view	Generative	3D Diffusion	GS	Real-time

Table 2. Table 2 : Quantitative results on full images from CO3D. We compare with other methods that also perform single-image-input novel view synthesis on full images, rather than only on masked objects. Our model outperforms other real-time rendering baselines. † \dagger : cited from [ 65 ] ; ⋆ \star : trained by us. The reported inference times account for the complete pipeline.

Rendering	Methods	Time Cost / Sequence		Hydrants				Teddybears
Speed		Inference	Scene Render	PSNR ↑	LPIPS ↓	FID ↓	KID ↓	PSNR ↑	LPIPS ↓	FID ↓	KID ↓
Offline	PixelNeRF [78] $†$	5.0 sec	4.1 min	17.93	0.54	180.20	0.14	-	-	-	-
	DFM [65] $†$	2 min	8 min	17.47	0.42	84.63	0.05	-	-	-
	Splatter Image (Full Images) $\star$	49.4 ms	109.5 ms	17.37	0.492	155.4	0.127	\ul17.44	\ul0.478	102.8	0.069
	Ours - AutoReconstructor $\star$	15.7 ms	29.1 ms	\ul17.59	\ul0.471	\ul133.0	0.111	17.49	0.477	95.1	0.058
	Ours - Diffusion single sample $\star$	2.7 sec	29.1 ms	17.40	0.473	133.8	0.114	16.77	0.480	\ul97.8	\ul0.061
Real-time	Ours - Diffusion 20-best oracle $\star$	54.0 sec	29.1 ms	17.74	0.466	132.3	0.114	17.08	0.474	96.6	0.062

Table 3. Table 3 : RealEstate10K Results. We report the number of test scenes and resolution following the settings used in prior work to ensure fair comparison. The baseline results are cited from the corresponding published papers. † \dagger : cited from [ 65 ] ; “–”: Results are not reported.

Methods	Rendering speed	# Scenes	Resolution	PSNR ↑	LPIPS ↓	FID ↓	KID ↓
pixelNeRF [78] $†$	Offline	100	128	-	-	195.4	0.14
SparseFusion [84] $†$	Offline	100	128	-	-	99.44	0.04
DFM [65]	Offline	100	128	-	-	42.84	0.01
Ours Diffusion	Real-time	100	128	19.9	0.238	\ul49.83	0.01
ZeroNVS [55]	Offline	6,473	256	13.5	0.414	-	-
Ours Diffusion	Real-time	6,473	128	20.2	0.253	15.53	0.01
Ours Diffusion	Real-time	6,473	256	\ul19.6	\ul0.365	33.44	0.03

Table 4. Table 4 : Ablation studies on the Hydrants scenes from the CO3D dataset evaluated on Full Images .

Encoder-Reconstructor Variants	PSNR ↑	LPIPS ↓	FID ↓	KID ↓
Splatter Image	17.37	0.492	155.4	0.127
No skip connections	16.63	0.551	167.9	0.143
Skips from first 2 layers	16.86	0.554	163.3	0.135
One Gaussian splat per pixel	17.43	0.491	140.1	0.115
No variational sampling	8.30	0.726	290.1	0.293
No SSIM	17.34	0.505	181.1	0.162
No LPIPS	\ul17.51	0.502	149.7	0.118
No Color Augmentation	17.20	\ul0.486	132.4	0.108
Our AutoReconstructor	17.59	0.471	\ul133.0	\ul0.111

Table 5. Table C1 : Availability of an open-source implementation for different related methods. We choose our baselines from the closest competitors that provide open-source code and support scene-level reconstruction. *: Diffsplat is an object-centric method; † \dagger : Only an incomplete unofficial version is available. Code availability checked on August 15, 2025.

Methods	Open-source Code	Baselines Used
DFM [65]	✓	✓
SplatterImage [61]	✓	✓
pixelNeRF [78]	✓	✓
SparseFusion [84]	✓	✓
ZeroNVS [55]	✓	✓
LGM [64]	✓	✓
Diffsplat [39] *	✓	$\times$
SampleSplat [23]	$\times$	$\times$
Bolt3D [62]	$\times$	$\times$
GRM [75]	$\times$	$\times$
GS-LRM [80] $†$	$\times$	$\times$
Zero-1-to-G [45]	$\times$	$\times$
Xiang et al. [72]	$\times$	$\times$

Table 6. Table D1 : Quantitative results on masked objects from the CO3D dataset. To compare against baselines which only predict novel views for the masked foreground object, as shown in the “Splatter Image” row in Fig.  D3 , we masked the background areas and compare metrics. † \dagger : cited from respective papers or [ 65 ] ; ⋆ \star : trained by us.

Methods	Hydrants				Teddybears
	PSNR ↑	LPIPS ↓	FID ↓	KID ↓	PSNR ↑	LPIPS ↓	FID ↓	KID ↓
pi-GAN [4] $†$	-	-	92.1	0.080	-	-	125.8	0.118
EG3D [5] $†$	-	-	229.5	0.253	-	-	236.1	0.239
GET3D [18] $†$	-	-	303.3	0.380	-	-	244.5	0.280
HoloDiffusion (no bootstrap) [32] $†$	-	-	277.9	0.305	-	-	222.1	0.217
HoloDiffusion [32] $†$	-	-	100.5	0.079	-	-	109.2	0.106
pixelNeRF [78] $†$	21.76	0.203	-	-	19.38	0.290	-	-
Splatter Image (Objects Only) [61] $†$	21.80	0.150	-	-	19.44	0.231	-	-
Splatter Image (Full Images) $\star$	24.00	0.141	75.22	0.045	\ul22.80	0.172	44.93	0.026
DFM [65]	23.37	0.133	60.1	0.029	-	-	-	-
Our AutoReconstructor $\star$	\ul24.18	0.128	49.45	0.025	22.88	\ul0.176	\ul47.54	0.026
Ours Diffusion single sample $\star$	23.87	\ul0.127	\ul49.34	\ul0.024	22.01	0.179	48.17	0.026
Ours Diffusion 20-best oracle $\star$	24.36	0.124	48.12	0.022	22.42	\ul0.176	47.62	0.027

Equations

$$\mathbf{h}={\mathbf{h}}_{\mu }+ϵ{\mathbf{h}}_{\sigma },\text{where }ϵ\sim \mathcal{N}(0,\mathrm{I}).$$

$${\mathcal{L}}_{rec}=\sum _{I\in {\mathcal{I}}^{+}}{\lambda }_1||I-\hat{I}|{|}^2+{\lambda }_2\mathrm{SSIM}(I,\hat{I})+{\lambda }_3\mathrm{LPIPS}(I,\hat{I})$$

$${\mathcal{L}}_{KL}=\mathrm{KL}\left(\mathcal{N}({\mathbf{h}}_{\mu },{\mathbf{h}}_{\sigma }\mathrm{I}),\mathcal{N}(0,\mathrm{I})\right).$$

$${\mathbf{h}}_t=\sqrt{{\bar{\alpha }}_t}\mathbf{h}+\sqrt{1-{\bar{\alpha }}_t}\mathbf{ϵ},$$

$${\mathbb{E}}_{\mathbf{h},\mathbf{ϵ}\sim \mathcal{N}(0,\mathrm{I}),t\sim \mathcal{U}(T)}||\mathbf{ϵ}-{\mathbf{ϵ}}_{\theta }({\mathbf{h}}_t,{\varphi }^{\text{ref}},t)|{|}_2^2,$$

Full text

1 Introduction

Gaussian splatting [34] has demonstrated impressive performance in modeling 3D scenes, delivering high-quality renderings with applications in augmented reality, world simulations, and robotics. Traditionally, dense observations, typically hundreds of RGB images, are required to accurately capture scene details and constrain the reconstruction process [46, 34], due to the large number of unknown parameters. However, such dense observations are not always available, often resulting in blurriness, artifacts, or empty regions caused by underconstrained reconstructions in occluded or invisible areas, such as the backs of objects and regions outside the camera frustum. In cases where sparse or even a single view is available, prior knowledge beyond the observations becomes critical for constraining the reconstruction.

We aim to address the learning of prior knowledge from large-scale datasets to help constrain the highly ill-posed problem of reconstructing a complete 3D scene from a single RGB image. Researchers have trained neural networks to reconstruct 3D Gaussian splats from a single image [61, 60] or sparse two-view images [7, 8]. These approaches can be broadly categorized into two formulations: regression-based and generative. As regression-based methods, Splatter Image [61] and its follow-up works [60, 7, 8] predict multiple Gaussians along each camera ray in a pixel grid using a single forward pass, efficiently modeling scenes within the image frustum. However, a fundamental limitation exists for regression-based formulations. They produce unimodal predictions, where only a single output is predicted for a given input image. This forces the model to average multiple hypotheses into a single “mode” for occluded or ambiguous regions, often resulting in blurry or inaccurate reconstructions. Due to this limitation, Splatter Image demonstrates performance only on isolated foreground objects. While follow-up works [60, 7, 8] extend these methods to scene-level reconstruction, they remain constrained to simpler tasks such as interpolation and exhibit limited extrapolation ability, particularly when generating parts of the scene that are not directly visible.

Instead, we propose a generative approach to learn a distribution over 3D Gaussian splats, enabling the sampling of multiple high-quality hypotheses in ambiguous situations, as shown in Figure 1. Generative formulations have also been explored on NeRFs [65] and Gaussian splats [70, 23]. Different from LatentSplat [70], which follows a variational autoencoder (VAE), we leverage more powerful denoising diffusion models [25], which have demonstrated a stronger ability to model complex distributions in images [53] and 3D models [31], to better capture the complex distributions of Gaussian splatting.

Recently, diffusion models have been used for Gaussian splatting. Some approaches [9, 77, 63, 36] leverage priors from 2D image diffusion models to generate 3D Gaussians, which requires intensive optimization and can suffer from consistency issues. Limited work exists that learns diffusion models to directly model 3D distributions of splats. For example, Bolt3D [62] relies on extra regression heads for generating Gaussians from denoised intermediate outputs. Some methods require ground truth 3D data of Gaussian splats [72, 45], which is expensive to obtain and limits scalability to diverse scenes. DFM [65] trains a denoising diffusion model over NeRFs [46] from only image supervision with forward models. Its architecture requires computationally expensive denoising for each rendered image and also inherits NeRF’s high computational cost.

Different from the above, we propose a novel generative method for 3D Gaussian splatting based on a latent diffusion architecture [53] to directly learn the distribution of 3D Gaussian splats in a latent space from only image supervision, enabling both real-time computation and high-quality, diverse samplings. This is not a straightforward process, and we make the following critical contributions to enable an effective training pipeline. The first challenge is how to acquire 3D ground truth Gaussian splat training data. To address this, we propose a novel method, Variational AutoReconstructor, to learn a latent space for 3D Gaussian splats using self-supervised losses derived solely from images. To mitigate the loss of high-frequency details in the latent space, we use skip connections to propagate and preserve fine-grained details. We further propose an approach based on classifier-free guidance to provide flexible control that balances faithfulness and randomness in the outputs for different applications. Our contributions are as follows:

• •

We propose a generative Gaussian splatting reconstruction method with a latent diffusion model for generating complete 3D scenes from a single RGB image, enabling the completion of occluded surfaces with multi-view consistency and efficient reasoning.

• •

We present a novel Variational AutoReconstructor to learn the latent space for Gaussian splats from only 2D posed images using self-supervised differentiable rendering losses.

• •

We present flexible control over the fidelity of reconstructed visible surfaces and the diversity of generated occluded surfaces within a latent diffusion model, using skip connections and classifier-free guidance.

Our experiments on the Hydrants and TeddyBears categories from the CO3D [52] dataset, as well as room scenarios from the RealEstate10K [83] dataset, show that our predictions are sharper and more complete than those of state-of-the-art methods, and provide new abilities to sample diverse 3D outputs.

2 Related Work

There is limited work on training denoising diffusion models to directly output 3D Gaussian splats. We provide an overview highlighting the differences with the broader topic in Table 1, and discuss them in detail below.

Summary

Unlike novel view synthesis methods that produce only 2D images [20, 6, 27], or videos lacking explicit 3D modeling and consistency [26, 22], or virtual scenes generated by text-to-3D models that suffer from domain gaps compared to real scenes [77, 63], our model naturally learns and outputs distributions of 3D Gaussian splats of real scenes with guaranteed 3D geometric consistency. Unlike object-only reconstruction models [73, 79, 2, 84], we reconstruct complete 3D scenes, including both foreground objects and background. Different from offline NeRF-based methods [65], our representation can be rendered in real-time. And unlike regression-based methods [61, 7, 8], we use a generative formulation with a denoising diffusion model, enabling the completion of occluded or invisible parts and the sampling of diverse scene variations.

2.1 Learning 3D Reconstruction and Generation

Object-centric Models [73, 79, 2, 84, 76, 41, 75, 58] reconstruct and generate 3D object models given text prompts or images, but lack awareness of background and scene context. Due to the domain gap in large-scale object datasets [14], the generated outputs often appear synthetic and virtual, with noticeable discrepancies from real-world textures and geometry. Notably, DMV3D [76] denoises a triplane NeRF representation, whereas ours denoises Gaussian splatting in a latent space, enabling much faster inference and training. Most importantly, it remains unclear how well these methods scale to larger and more complex scene-level reconstruction tasks, especially in the absence of large-scale, ground-truth 3D scene data.

Scene-level “Regression” Models predict a single “mode” of the 3D representation from partial input data. However, in ambiguous situations, such as occluded areas, the network tends to learn an “average” mode, which often leads to blurry results. Early works focused on voxel grids [71, 59, 68, 12, 13, 11, 17] and occupancy fields [29], which model geometry only, without capturing texture. Dust3R and its variants [67, 66] show promising end-to-end learning for camera localization and reconstruction with point clouds, which is orthogonal to ours, where we focus on representing 3D scenes. To synthesize 360∘ views with NeRFs [46], PixelNeRF [78] predicts novel views conditioned on features extracted from single (or few) input images. NViST [30] further extends this with a Transformer to predict novel views. However, these methods are limited by the slow rendering process of NeRF, which is mitigated by Gaussian splats [34]. Splatter Image [61] and follow-up works [15, 28] predict multiple Gaussians along camera rays in a pixel grid, resulting in efficient methods to model scenes within image frustums. Multi-view inputs [7], depth priors [60, 43, 74], and structural constraints [8] have been further explored to regularize the outputs.

Scene-level “Generative” Models can predict a “distribution” of 3D scene representations to model complex and ambiguous situations. LatentSplat [70] follows a variational autoencoder (VAE) design to infer the distribution of Gaussian splats. We leverage more powerful denoising diffusion models [25], which have demonstrated a stronger ability to model complex distributions in images [53] and 3D models [31], to better capture the complex distributions of Gaussians. DFM [65] trains a denoising diffusion model to learn a distribution of NeRFs conditioned on a single input image. However, it requires running denoising steps for each novel view, requiring extremely intensive computation.

Novel View Synthesis and Video Generation Models output images or videos [44, 37] conditioned on camera poses. NerfDiff [20], GeNVS [6], and ViewDiff [27] train diffusion models to generate novel views. However, they cannot directly output a coherent 3D representation. Instead, they need to run the intensive denoising process separately for novel views to generate images or videos, which increases computational cost and, most importantly, limits multi-view consistency without 3D geometry.

2.2 Diffusion Models for Gaussian Splats

2D Diffusion: Lifting Image Diffusion Models for 3D

2D image generation models [53] contain priors that can be used to optimize 3D representations using Score Distillation Sampling (SDS) from rendered multi-view images of NeRFs [50, 55] or Gaussian splats [9, 77, 63, 36]. Some approaches first synthesize multiple novel view images and then optimize the 3D scene [16, 42, 19]. However, a large number of novel views are required to fully constrain 3D representations during optimization, where identity and multi-view consistency are challenging, requiring finetuning of the large diffusion model (e.g. Zero 1-to-3 [42] and Cat3D [19]). Some methods [82, 57] use video diffusion models as priors. In principle, the priors of image and video generation models operate in 2D image space rather than directly on the 3D representation. As a result, these methods suffer from long optimization times and geometric consistency issues, such as the multi-face problem [50].

3D Diffusion: Direct Modelling of 3D Gaussian Splats Distributions

There is limited work exploring diffusion models to directly learn the distribution of 3D Gaussian splats, with the biggest challenge being the lack of large-scale 3D ground-truth Gaussian splat datasets for real scenes. Some works focus on objects [45, 31, 47, 38] rather than entire scenes. Some works finetune diffusion models originally designed for image generation [72, 39, 45], tailoring the output to align with Gaussian splat modalities. However, [72, 39, 45] require ground truth 3D Gaussian splats in order to learn the latent space, making these methods computationally intensive due to the need to build a dataset in advance. In contrast, we learn the latent space directly from images without requiring ground truth splats, with the potential to scale up with the availability of large-scale image datasets. Bolt3D [62] uses a multiview diffusion model to generate images and requires an additional intermediate model to produce a splat representation. Some works [49] rely on an extra regression-based model as a teacher to supervise the diffusion model, complicating the process. Most similar to ours, Henderson et al. [23] train a denoising diffusion model to generate Gaussian splats directly. Their model is trained to encode multi-view input images into a multi-view latent space. However, during inference, it requires knowledge of multiple camera positions, which limits its applicability in real-world scenarios, whereas ours only requires single-view input.

3 Method

In this section, we first discuss the challenges of training denoising diffusion models on 3D Gaussian splats in the absence of ground-truth 3D data. We then propose a novel architecture, the Variational AutoReconstructor, which learns a latent space using forward models (differentiable renderings), along with critical modifications for preserving high-frequency details. These components together enable the training of a diffusion model to generate 3D Gaussian splats.

3.1 Training Diffusion Models without Ground Truth Samples

Conventional denoising diffusion models [25, 53] require ground-truth samples to which noise is progressively added, and from which the denoiser learns to remove noise. We follow Splatter Image [61] to model 3D Gaussian splats within an image frustum by predicting multiple Gaussians along each camera ray. In order to train a diffusion model to learn a distribution $P(X)$ over the Splatter Image $X$, we need access to a large-scale dataset of $X$, i.e., ground-truth Splatter Images. However, unlike images [53], videos [26], or 3D objects [31], no large-scale scene-level datasets exist for 3D Splatter Image representations, which are costly and time-consuming to construct. One approach might be to pre-optimize Splatter Images from multi-view inputs [72]. However, aside from the computational and scalability limitations, our early experiments with this method revealed sparsity and continuity issues hinder the effective learning of neural networks. Therefore, we propose a more scalable method that directly learns from image datasets [52, 10, 83], without relying on intermediate modules.

Training Diffusion Models over Gaussian splats from Images Supervision

The feasibility of training diffusion models from low-dimensional observations has been scarcely explored in the literature. Theoretically, our goal is to learn a diffusion model that approximates the distribution $P(X)$ without access to ground truth samples of $X$, but only to indirect projections $f(X)$ of $X$. Here, $f$ is a differentiable function—referred to as a “forward model” or “observation model”, which outputs a lower-dimensional partial projection of the high-dimensional variable $X$. In our case, the projections are rendered images from Gaussian splats. This links our image datasets of scenes to the potential unknown Gaussian representations that can model those scenes. Instead of directly training on Splatter Images, we follow a latent diffusion architecture, which models the distribution over the latent space. The latent space learns and compresses the 3D structure of Splatter Images and reduces their high-dimensional parameters, enabling much more computationally efficient training of the diffusion model.

3.2 Learning a Latent Space of 3D Splatter-Images from 2D Images

We propose a novel architecture, Variational AutoReconstructor, to learn the latent space in a self-supervised manner from only images, over which a denoising diffusion model is then trained. We discuss alternative methods that did not work in our early experiments in the Sup. Mat. A.3.

Variational AutoEncoder (VAE)

The conventional latent diffusion pipeline [53] involves learning a latent space with a Variational Autoencoder (VAE) [35], as shown in Figure 2(a), where an encoder $q_{\phi}$ takes a ground-truth sample $X$ as input and maps it to a lower-dimensional latent distribution $q_{\phi}(Z|X)$. A latent is sampled and decoded back to $X^{\prime}$ using a decoder $p_{\theta}$. A loss is computed between $X$ and $X^{\prime}$ to ensure faithful reconstruction of the data, and this loss is used to train both the encoder $q_{\phi}$ and the decoder $p_{\theta}$. However, the conventional latent space learning is not applicable in our case, because we do not have access to the ground-truth Splatter Images $X$, but only to the images $m=f(X)$, which are projections of $X$.

Variational AutoReconstructor (VAR)

To address this gap, we propose a new architecture called the Variational AutoReconstructor (VAR), as shown in Figure 2 (b). This encodes a 2D image $m=f(X)$—a projection of $X$—into a latent distribution $q_{\phi}(Z|m)$, and then reconstructs a sample back to a high-dimensional representation $X^{\prime}$, which corresponds to the unknown 3D Splatter Image representation. We supervise $X^{\prime}$ using posed multi-view images $\{m_{i}\}$, each of which is a projection of $X$, available in large-scale image datasets. The differentiable function $f(\cdot)$ makes the entire pipeline end-to-end trainable.

Figure 3 gives an overview of the Variational AutoReconstructor training pipeline. We assume access to training datasets containing multiple image sequences capturing different object instances or scenarios, such as CO3D [52] and RealEstate10K [83]. Each training example comprises a reference image $I^{\text{ref}}$, a set of target images $\mathcal{I}=\{I^{\text{tgt}}_{i}\mid i=0,\ldots,t\}$, and the corresponding camera poses and intrinsics. At training time, the reference image $I^{\text{ref}}$ is input to the encoder, which produces a Gaussian distribution represented by a mean $\mathbf{h}_{\mu}$ and variance $\mathbf{h}_{\sigma}$. The latent code $\mathbf{h}$ is sampled from the Gaussian distribution $\mathcal{N}(\mathbf{h}_{\mu},\mathbf{h}_{\sigma}\mathrm{I})$ using the reparameterization trick [35]:

[TABLE]

This sampled latent code is passed to the reconstructor to obtain the Splatter Image representation corresponding to the reference view. The predicted Splatter Image is then converted into a Gaussian splat representation of the scene and rendered into the target views using the known camera intrinsics and poses for each view $I_{i}$ in $\mathcal{I}$, denoted as $\hat{I_{i}}$. In practice, we also render the reference view.

Preserving High-frequency Details with Skip connections

We observe that a Splatter Image directly reconstructed from the low-dimensional latent space captures the geometry but loses high-frequency texture details. To overcome this, we introduce a skip connection [54], which allows high-frequency information to flow directly from the encoder’s first-layer features to the higher-resolution layers of the reconstructor, as demonstrated in Figure 4. At test time, we gain flexibility by modifying the weight of skip connection features to control the faithfulness of the reconstruction to the input images, potentially balancing the diversity of reconstructions. More results are in the supplementary material section D.2.

Losses

The reconstruction loss $\mathcal{L}_{rec}$ for a single training example is:

[TABLE]

where $\mathcal{I^{+}}=\{I^{ref}\}\cup\mathcal{I}$ includes the reference and target images and $\lambda_{*}$ are weights for the different loss terms. SSIM [69] encourages similarities in structure and contrast. LPIPS [81] matches neural network encodings of image patches, which encourages photorealism. Following VAE [35], we also use a KL divergence loss $\mathcal{L}_{KL}$ on the latent space $z$ to regularize it to follow a prior Gaussian distribution:

[TABLE]

3.3 Training the Latent Diffusion Model

Once the Variational AutoReconstructor is trained, we encode all training images into latent code samples. A denoising diffusion model is then trained to learn the distribution of these latent codes conditioned on the input. The overall pipeline is illustrated in the supplement Fig. A1.

In general, we follow previous latent diffusion training pipelines [53, 33]. The forward diffusion process for latent code $\mathbf{h}$ at time $t$ is

[TABLE]

where $\boldsymbol{\epsilon}~\sim~\mathcal{N}(0,\mathrm{I})$ is the sampled Gaussian noise, $\bar{\alpha}_{t}=\prod_{s=1}^{t}{(1-\beta_{s})}$ is the noise variance and $\beta_{1},\beta_{2},\dots,\beta_{T}$ is the variance schedule [25]. During training we minimize the denoising loss:

[TABLE]

where $\boldsymbol{\epsilon}_{\theta}(\mathbf{h}_{t},\phi^{\text{ref}},t)$ is the denoising diffusion model that takes the noised latent code $\mathbf{h}_{t}$, input image features $\phi^{\text{ref}}$ and time step $t$ as input.

Conditioning Diffusion Models on Input Images

Conventional diffusion models [53] take text prompts as conditions, which are encoded into feature vectors. Instead, we condition on the input reference image $I^{\text{ref}}$, preserving the image structure and correspondence, to generate a Splatter Image. Following Marigold [33], we concatenate features $\phi^{\text{ref}}=F_{feat}(I^{\text{ref}})$ extracted using a feature encoder $F_{feat}(.)$ from the input image $I^{\text{ref}}$ with the noised latent codes as input to the denoiser. There are several options for the feature encoder $F_{\text{feat}}(\cdot)$. We directly use the pre-trained encoder from the Stable Diffusion VAE model [53] to obtain the conditional features $\phi^{\text{ref}}$. As future work, we plan to explore other embeddings such as CLIP [51] to condition on text prompts.

Controlling Diversity with Classifier-Free Guidance

To increase the diversity of samples for generation tasks, we train with classifier-free guidance [53], where conditioning features are zeroed out 20% of the time (akin to unconditional generation). This reduces overfitting to the scene appearance visible in the conditioning view. During inference, it provides the option to increase the guidance weight to generate more diverse samples to balance between faithfulness and diversity.

3.4 Diffusion Inference

During inference, our method takes a single image as input and outputs a Gaussian splat represented as a Splatter Image (Fig. A2, in Sup. Mat). The input image is first encoded by a feature encoder to generate conditioning features. A randomly initialized latent code is concatenated with these features and fed into the denoising diffusion model for 50 diffusion steps. The resulting denoised latent code is then decoded into a Splatter Image using the Reconstructor trained in the first stage. This Splatter Image is subsequently backprojected to 3D to produce the final Gaussian splat.

Controlling Faithfulness and Diversity

As a generative method, our model enables sampling multiple outputs by denoising different randomly initialized latent codes. The trade-off between fidelity to the input image and diversity of the generated results can be controlled by adjusting the weights of classifier-free guidance and skip connections.

4 Experiments

We introduce the basic experimental settings below and provide implementation details in the Supplementary Material.

4.1 Experimental Settings

Datasets

We train and evaluate our model on the challenging CO3D dataset [52] and the RealEstate10K dataset [83]. The CO3D dataset comprises 360-degree video captures of various objects in both indoor and outdoor real-world contexts, with annotated camera poses obtained via Structure-from-Motion (SfM) [56]. Following prior works [61, 70, 47], we train and evaluate on the most common categories, Hydrants and Teddy Bears, which include 723 and 1329 scenes, respectively. The RealEstate10K dataset [83] contains real-world videos of residential indoor and outdoor environments, with camera poses also derived using SfM. We use the provided official training and testing splits.

Baselines

We compare our method with several state-of-the-art baselines, including: (1) Original Splatter Image [61], a forward method using Gaussian splatting, applied only on objects; (2) Splatter Image (Full Images)$\star$, which we train on full images including the background; (3) PixelNeRF [78], a forward method based on NeRF; (4) DFM [65], a diffusion-based NeRF model; (5) ZeroNVS [55], a novel view synthesis method; (6) SparseFusion [84], which optimizes a scene using score distillation for each view; and (7) LGM [64]: A large-scale object reconstruction model based on Gaussian splats. We introduce the baselines in detail in the supplementary material.

Metrics

Following prior work [78, 40, 61], we report PSNR and LPIPS scores, where PSNR quantifies pixel-wise reconstruction fidelity and LPIPS measures perceptual similarity based on deep features between rendered and ground-truth images. Following [27], we report FID [24] and KID [1] scores for generative performance, which measure image similarity at the distribution level by comparing statistics of deep image features over multiple images, thereby evaluating visual realism. We further report scores separately on Full Images, which computes metrics on all pixels of reference views, and Objects Only, which computes metrics only on object pixels following [61], for a fair comparison with prior object-centric work.

4.2 Reconstruction Performance

CO3D Datasets

We present quantitative results on full images in Table 2. The results show that our method outperforms real-time baselines on most metrics. Although DFM [65] outperforms our method on some metrics, it is considerably slower due to its NeRF representation and per-frame diffusion design. PixelNeRF [78] achieves slightly higher PSNR than our method but performs significantly worse in LPIPS, likely because it produces generic smooth details in unobserved regions. As shown in Fig. 5 and Fig. 6, our results appear considerably sharper than those of PixelNeRF and SplatterImage, and are comparable to DFM, while being much faster to compute (3 seconds versus 10 minutes). We output real-world scenes with both foreground and background, in comparison to object-only models such as LGM [64]. We provide a detailed quantitative evaluation of object reconstruction methods in Table D1 in the Supplementary Materials.

RealEstate10K Datasets

We present quantitative results on the RealEstate10K dataset in Table 3. To fairly compare with baselines that follow different evaluation settings, we follow PixelNeRF [78], SparseFusion [84], and DFM [65] by evaluating on 100 scenes at a resolution of $128\times 128$. We also follow ZeroNVS [55] to evaluate on all 6,473 scenes and report results at both $128\times 128$ resolution and $256\times 256$, with the latter obtained by upsampling our rendered images. Our method outperforms SparseFusion, PixelNeRF, and ZeroNVS by a significant margin, as we generate a consistent 3D representation. In contrast, SparseFusion and ZeroNVS suffer from view inconsistency due to their reliance on multi-view optimization, and the intensive computation. We also achieve performance comparable to DFM while being significantly faster at inference time. We show qualitative results compared to DFM in Figure 7, where we achieve better performance in some challenging regions.

4.3 Generative Performance

Compared with regression-based baselines, our method can generate diverse samples with varying color, structure, and style for unobserved and uncertain parts, as shown in Figure 1. We demonstrate this ability in detail below.

Qualitative Results

As shown in Figure 8, our diffusion model is able to sample diverse results in ambiguous situations where parts of objects are occluded with black boxes. The filled-in areas exhibit reasonable 3D structure consistent with the visible parts and remain consistent as part of a holistic 3D representation when viewed from novel perspectives. This demonstrates that the model effectively learns 3D priors purely from 2D images. We further demonstrate this in Figure D2 in the Supplementary Materials, where, compared with DFM, we are able to control the diversity in the occluded back of objects, showing increasingly strong differences to balance fidelity to the input image against output diversity.

Quantitative: $k$-best oracle evaluation

Generative models are capable of producing multiple possible predictions. Oracle $k$-best evaluation [23, 21, 3] allows the model to produce $k=20$ samples for each input, and the score of the best sample is used to evaluate that input image. This metric shows an upper bound reconstruction performance, assuming some manual effort can be made to select the best result. It also reflects the generative model’s ability to capture the distribution and produce diverse samples. The results of our diffusion model on the CO3D datasets are shown in Table 2 and the Sup. Mat.. We observe an improvement in the reported metrics compared to single samples, which demonstrates that our model generates diverse outputs rather than collapsing to a single mode.

4.4 Ablation Studies

We ablate the key modules and hyperparameters of our AutoReconstructor to validate our contributions in Table 4. Our method scores highest or second highest across all metrics, validating our design decisions. The “No Augmentation” row shows that our augmentations boost PSNR and LPIPS at the expense of a small drop in FID and KID scores. More discussions are given in the Supplementary Materials.

5 Conclusion

We presented a training pipeline for a denoising diffusion model that reconstructs complete 3D scenes using Gaussian splats from a single RGB image as input. By proposing a Variational AutoReconstructor architecture, we are able to efficiently learn a latent space for Splatter Images using only 2D images, from which a denoising diffusion model can be trained by conditioning on input image features. Experiments on the CO3D and RealEstate10K datasets demonstrate that our approach achieves state-of-the-art results in scene reconstruction, particularly in occluded regions. Furthermore, our method enables diverse sampling of Gaussian splats with controllable trade-offs between faithfulness to the input and generative diversity.

Appendix A Methods Details

A.1 Gaussian Splatting Representation

We represent scenes as a set of Gaussian splats [34], and follow Splatter Image [61] to represent Gaussians inside a frustum. In this section, we first introduce the definitions of Gaussian splatting, then describe how to represent Gaussians within an image frustum, and how a neural network is used to predict them.

Gaussian Splatting

Following [34], a 3D scene is parameterized as a set of Gaussian splats, each of which is associated with an opacity $\sigma$, 3D position $\mu$, scale and rotation parameters of the covariance matrix, and color $c$ modeled with spherical harmonics. Through differentiable rendering, all Gaussians can be projected onto image planes to form RGB images. Typically, the parameters of the Gaussian splats are optimized to match the renderings with a set of posed RGB training images, in order to reconstruct a single scene at a time.

Splatter Image

The “Splatter Image” [61] formulation predicts a Gaussian for each pixel ray of a single $H\times W$ input RGB image $I$, representing both observed and unobserved scene content inside the image frustum. This results in a parameter matrix of shape $H\times W\times N$, where $N$ is the number of parameters per Gaussian splat. Specifically, the Gaussian mean is parameterized by the pixel depth and a 3D offset. Each Gaussian thus contains a total of $N=15$ scalar values: 1 (opacity), 1 (depth), 3 (offset), 3 (scale), 4 (rotation quaternion), and 3 (color). We use only the DC coefficients for spherical harmonics. To further encourage the network to model occluded surfaces, follow-up work [60] extends the parameter matrix to $H\times W\times MN$, where $M$ denotes the number of Gaussian splats predicted per pixel, i.e., the number of Gaussian layers.

Training a Network to Predict Splatter Image

Following [61], a neural network can be trained to directly predict the Gaussian splat parameters of a Splatter Image via a feed-forward process. Given a single input image, the network (e.g., a U-Net) outputs the complete parameter matrix corresponding to the Gaussian splats, while preserving the spatial structure of the image. However, this network is conventionally regression-based and can only model a single possibility [61, 7, 60]. In contrast, we extend this formulation to a distribution using a diffusion model, enabling the modeling of multiple plausible outputs.

A.2 Diffusion Pipelines

To better illustrate our diffusion architecture, we present figures for diffusion training over the learned latent space in Figure A1, and the diffusion inference procedure in Figure A2.

A.3 Alternative Approaches for Learning a Latent Space for Splatter Images

Our Variational AutoReconstructor learns a latent space of Splatter Images in a simple and effective way. We further describe alternative approaches we explored in the early stages of the project that did not yield satisfactory results.

Reconstructor-only without the Encoder

Since we only need latents for diffusion training, we try removing the encoder. We assign each image an optimizable latent code and optimize both the latent code and the reconstructor weights to generate Splatter Images, similar to the autodecoder approach for SDF representations [48]. However, we observe that this leads to low-quality reconstructions because the optimization gets stuck in the early stages and fails to capture the geometry correctly. We assume the encoder is important for initializing the latents or capturing correlations that are crucial to the input images.

Pre-optimizing 3D Gaussian splats and projecting into Splatter Images

This approach optimizes 3D Gaussian splats for each scene using all available images, then projects the Gaussians onto each image plane to form Splatter Images. However, rasterization errors occur when projecting Gaussians to achieve pixel alignment, and the 3D Gaussians are not distributed uniformly across the images. The resulting Splatter Images are extremely sparse, with 80% of the pixels lacking a corresponding Gaussian. We find it difficult to train a VAE to encode each modalities of such sparse splatter images and reconstruct them with negligible error, which breaks the geometric and texture consistency.

Appendix B Implementation Details

B.1 Network architecture

We implement the AutoReconstructor architecture following the VAE from Stable Diffusion [53]. The encoder remains unchanged and predicts a distribution over the latent representation at $\tfrac{1}{8}$ the resolution of the input images, with $4+4$ channels (4 for $\mathbf{h}_{\mu}$ and 4 for $\mathbf{h}_{\sigma}$), from which a 4-channel latent can be sampled. We extend the decoder into a reconstructor by modifying its output layers: instead of producing 3 channels for RGB images, it now outputs $MN$ channels to represent Gaussians for each pixel. We predict $M=2$ Gaussians for each pixel, and set $N=15$ as described in Section A.1. We train the AutoReconstructor by fine-tuning it based on the original VAE parameters. We follow the same denoiser architecture as Stable Diffusion [53] and train it from scratch.

B.2 Data Preprocessing

For a fair comparison with prior work [6, 61, 65], all images in the datasets are cropped around the principal point of the camera (using the largest possible crop) and rescaled to $128\times 128$ pixels before training.

Scaling of scenes

We follow the approach outlined in the Splatter Image [61] supplementary material (Section B.2) to address varying scene scales. In practice, this involves rescaling the predicted Gaussian depths (at both training and test time) to lie between $z_{\text{near}}$ and $z_{\text{far}}$, where $z_{\text{near}}$ and $z_{\text{far}}$ are computed using the known depth to the object in the scene.

B.3 Training Details

Encoder-Reconstructor

We first train the AutoReconstructor. We use batch size 32 and Adam optimizer. We train for 6000 epochs with L2 and SSIM losses ($\lambda_{1}=0.8,\lambda_{2}=0.2$) and learning rate of 5e-5. Fine-tuning is done with another 8000 epochs with L2, SSIM and LPIPS losses ($\lambda_{1}=0.8,\lambda_{2}=0.2,\lambda_{3}=0.01$) and learning rate of 5e-6. Using LPIPS loss only during finetuning is a common practice [61] to prevent the LPIPS loss from disrupting geometric structure learning in the early stages of training. We apply the same random color augmentation to both the reference and target views to increase the diversity of colors that the latent codes can represent.

Skip connection

When using the skip connection, we observed that the model can “cheat” by retaining most of the information about the scene in the skip connection features instead of latent codes. To limit this effect, during training we randomly zero out all the values of the skip features, encouraging the network to encapsulate scene information in the latent codes.

Denoising Diffusion Model

We train with Adam Optimizer and batch size 32. We train for $1,000$K steps and a probability of 20% for unconditional generation (classifier-free guidance). We finetune for $100$K steps with a probability of 50% for randomly masking the input images. Learning rate is set to 1e-4 with exponential learning rate scheduler, similar to Marigold [33]. We also sample latent codes from the encoder network as ground truth latent codes during diffusion trainings.

Appendix C Experiments Details

C.1 Evaluation Protocol

After training on the CO3D or RealEstate10K datasets, we evaluate on the unseen test sets. For each sequence, we use the reference view to predict Gaussian splats and render them from both the reference viewpoint and several novel target viewpoints. We then compare the rendered images with the corresponding ground-truth RGB images captured from the same viewpoints. We design two settings for comprehensive evaluations: Full Images and Objects Only. Full Images, computes the metrics on all pixels of reference views. The Objects Only evaluation computes the metrics only on object pixels, following the protocol of Splatter Image [61], and is reported here for a fair comparison with prior work. Similar to our method, DFM [65] and Splatter Image (Full Images)$\star$ always predict background pixels. For the Objects Only evaluation, we use the ground truth object mask to set the background pixels of both the predicted and ground-truth images to black before computing the metrics.

C.2 Baselines

Open-source methods

We present the open-source status of the literature methods in Table C1, from which we select our main competitors in the experiments for evaluation.

Chosen Baselines

We introduce in detail the baselines to which we compare below:

(1) Splatter Image [61] is the closest model to ours, employing a regression-based formulation. We use the publicly available code and models for our experiments. This model was trained only on foreground objects by masking out the background.

(2) Splatter Image (Full Images)$\star$: To obtain a baseline more comparable to our method, which models the full scene, we introduce a modified and retrained version of the original Splatter Image [61] that predicts both objects and background by training on full images. The $\star$ indicates that this model was trained by us. As a regression-based model, it produces uni-modal outputs and does not support sampling.

(3) PixelNeRF [78] is a NeRF-based regression method for novel view prediction conditioned on one or more images.

(4) DFM [65] is a diffusion-based method built on NeRF that can generate high-quality renderings. However, the average scene takes approximately 10 minutes to reconstruct and render. We use the official codebase to produce outputs for evaluation. It includes models trained on the Hydrants and RealEstate10K datasets.

(5) ZeroNVS [55] employs diffusion and Score Distillation Sampling with anchoring to optimize a NeRF, but it faces 3D consistency issues such as Multi-Face Janus artifacts [50].

(6) SparseFusion [84] outputs novel views of 3D scenes using a generative model to provide score distillation for each view, requiring intensive computation during optimization. We compare our method with ZeroNVS and SparseFusion on the RealEstate10K dataset.

(7) LGM [64]: A large-scale object reconstruction model based on Gaussian splats. We demonstrate our object reconstruction performance, while additionally reconstructing background regions.

C.3 Ablation Studies Analysis

The results of the ablation study are shown in Table 4, and we discuss them in detail below:

Skip connections

No skip connections: We ablate the variant that does not use skip connections, which also qualitatively shows degradation in Fig. 4. Skips from first 2 layers: We further ablate by using only two skip connections from the first and second layers of the encoder, which may reduce the scene information learned in the latent codes.

Number of Gaussians per Pixel

One Gaussian splat per pixel: Our method trained to predict only a single Gaussian splat per pixel, which lacks sufficient Gaussians to represent scene details.

Variational Sampling

No variational sampling: Disables variational sampling and directly uses the 4-dimensional code regressed by the encoder. We observe that training gets stuck and does not improve in the early stages.

Training Losses

We ablate training with No SSIM or No LPIPS, which results in critical performance drops.

Data Augmentation

No Color Augmentation: Does not apply color augmentation to the input and target images during training.

C.4 3D Generative Experiments

We present additional details of the 3D generative experiments in Figure 8. We train the diffusion model using random masking, while keeping the Variational AutoReconstructor model fixed. The masked input image is passed through the diffusion model to generate a latent code, which is then reconstructed into a Splatter Image using the pretrained AutoReconstructor. To focus on the generative capacity of the diffusion model, we disable skip connections during these experiments. Our goal is to demonstrate the model’s ability to produce diverse outputs from its latent space. To further enhance high-frequency details, techniques such as finetuning the AutoReconstructor with random masking could be explored, which we leave as future work.

C.5 Experiments Notes

Cited Baseline Scores

In the quantitative results, we cite the reported scores from the papers of the baselines and ensure that the settings are consistent for a fair comparison. The main baseline scores are reported in Splatter Image [61] and DFM [65]. Some combinations of training data and methods are not publicly available for citation or evaluation, e.g., the performance of the DFM model trained on TeddyBears is not reported, and the corresponding model is not publicly available. Thus, we leave the corresponding entries blank in the table.

Appendix D Additional Experiments and Analysis

D.1 Objects-level Evaluations

Results for Objects Only in Table D1 provide a broader comparison to methods focused solely on objects, where our method outperforms almost all of them. Splatter Image slightly outperforms on TeddyBear LPIPS and KID metrics, partly due to its object-oriented design, while our approach focuses on the whole scene. We present a qualitative evaluation on TeddyBears in Figure 6, where our method achieves more sensible reconstruction, especially in occluded areas.

We further compare our method with representative large-scale object reconstruction approaches. Some methods [75, 80] do not have official code, pretrained models, or output results available for comparison, as also noted in Table C1. We qualitatively compare with LGM [64] in Figure 6 and Figure D1, where our method demonstrates superior reconstruction quality, especially in preserving background context.

D.2 Additional Qualitative Results for Diverse Samplings

We show more diverse sampling results on Hydrants in Figure D2. For each input image, we show three generated reconstructions rendered from two different target views. From left to right, the diversity increases by manually increasing the classifier-free guidance weight and decreasing the skip connection weight. Specifically, the three samples are generated with skip connection weights of (1.0, 0.5, 0.0) and classifier-free guidance weights of (0.2, 0.5, 0.5), respectively. This controllability allows tailoring the output to specific application needs, favoring high fidelity for reconstruction tasks, and greater diversity for applications like data augmentation or creative generation.

In the figures, we also compare with DFM [65], which is a denoising diffusion model based on the NeRF representation, and show that our samples are much more diverse, especially in the occluded object areas. DFM shows small variations in the texture of objects, for example, some rust, while ours is able to add a valve structure and change the color. Furthermore, DFM denoises 2D images view by view, whereas ours outputs the Gaussian splats in a single diffusion process and is much faster.

D.3 Additional Qualitative Results for Reconstruction

• •

Figure D3, Figures D4 and D5 show more reconstruction results on hydrants and teddybears.

• •

Figure D6 further shows renders of depth maps from our Gaussian splats, demonstrating that our samples are diverse in both geometry and appearance.

D.4 Computation Analysis

On each dataset, the AutoReconstructor takes 6 days and the diffusion model takes 2 days to train, both on 1 A100 GPU. Compared with the closest diffusion model DFM, which requires 7 days of training on 8 A100 GPUs, our method is significantly more efficient. During inference, as shown in Table 2, generating a Splatter Image and rendering novel view images for one sequence takes only 3 seconds, while DFM requires 10 minutes. If using only the AutoReconstructor, the inference time can further decrease to 0.15 seconds. Overall, our pipeline is quite efficient in both training and inference, with strong potential to scale up to larger datasets and be used in online downstream applications.

Appendix E Limitations and Future Work

We make a step forward in training generative models on 3D scenes using Gaussian splatting representations. However, there are still limitations that suggest promising directions for future work. First, we model scenes using Splatter Images, which are largely constrained to the image frustum of a single view. Extending this representation to large-scale scenarios, either by exploring multiple Splatter Images or alternative representations that cover entire scenes, would be valuable. Second, large-scale training across diverse datasets, including both indoor and outdoor scenes and a wide range of object categories, could lead to a generalizable large reconstruction model. Finally, while we currently model static 3D scenes, extending the framework to 4D Gaussians to capture scene dynamics would enable broader applications in dynamic scene understanding and synthesis.