DrivingGaussian++: Towards Realistic Reconstruction and Editable Simulation for Surrounding Dynamic Driving Scenes

TL;DR

DrivingGaussian++ is a novel framework that enables realistic reconstruction and controllable editing of dynamic driving scenes, integrating scene modeling, editing capabilities, and LLM-based motion generation for enhanced scene diversity.

Contribution

It introduces a new scene modeling approach combining static and dynamic Gaussian representations with LiDAR priors, enabling realistic reconstruction and editing without training.

Findings

Outperforms existing methods in scene reconstruction and synthesis

Supports training-free controllable editing including texture, weather, and object manipulation

Generates diverse, realistic multi-view driving scenarios

Abstract

We present DrivingGaussian++, an efficient and effective framework for realistic reconstructing and controllable editing of surrounding dynamic autonomous driving scenes. DrivingGaussian++ models the static background using incremental 3D Gaussians and reconstructs moving objects with a composite dynamic Gaussian graph, ensuring accurate positions and occlusions. By integrating a LiDAR prior, it achieves detailed and consistent scene reconstruction, outperforming existing methods in dynamic scene reconstruction and photorealistic surround-view synthesis. DrivingGaussian++ supports training-free controllable editing for dynamic driving scenes, including texture modification, weather simulation, and object manipulation, leveraging multi-view images and depth priors. By integrating large language models (LLMs) and controllable editing, our method can automatically generate dynamic object motion trajectories and enhance their realism during the optimization process. DrivingGaussian++ demonstrates consistent and realistic editing results and generates dynamic multi-view driving scenarios, while significantly enhancing scene diversity. More results and code can be found at the project site: https://xiong-creator.github.io/DrivingGaussian_plus.github.io

Tables

Table 1. TABLE I : Overall perforamnce of DrivingGaussian++ with existing state-of-the-art approaches on the nuScenes dataset. Ours-S denotes the DrivingGaussian++ with SfM initialization, and Ours-L denotes training with LiDAR prior.

Methods	Input	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Instant-NGP [16]	Images	16.78	0.519	0.570
NeRF+Time	Images	17.54	0.565	0.532
Mip-NeRF [26]	Images	18.08	0.572	0.551
Mip-NeRF360 [27]	Images	22.61	0.688	0.395
Urban-NeRF [53]	Images + LiDAR	20.75	0.627	0.480
S-NeRF [34]	Images + LiDAR	25.43	0.730	0.302
SUDS [35]	Images + LiDAR	21.26	0.603	0.466
EmerNeRF [36]	Images + LiDAR	26.75	0.760	0.311
3DGS [14]	Images + SfM Points	26.08	0.717	0.298
Ours-S	Images + SfM Points	28.36	0.851	0.256
Ours-L	Images + LiDAR	28.74	0.865	0.237

Table 2. TABLE II : Editing Efficiency. We demonstrate the efficient multi-task editing capabilities of our method in comparison with existing state-of-the-art 3D editing approaches. The evaluated multi-task editing scenarios include object manipulation, weather simulation, texture modification, and dynamic editing. “Exec. Time” indicates the total execution time for completing each editing task under a fair comparison setting.

Method	Object Manipulation	Weather Simulation	Texture Modification	Dynamic Editing	Exec. Time	${\text{CLIP}}_{dir}\uparrow$
InstructNeRF2NeRF	\usym2717	\usym2713	\usym2713	\usym2717	$\approx$274 minutes	0.1570
InstructGS2GS	\usym2717	\usym2713	\usym2713	\usym2717	$\approx$60 minutes	0.0918
ClimateNeRF	\usym2717	\usym2713	\usym2717	\usym2717	$\approx$107 minutes	0.1105
Ours	\usym2713	\usym2713	\usym2713	\usym2713	$\approx$8 minutes	0.2327

Table 3. TABLE III : Overall perforamcne on the KITTI-360. Comparisions of DrivingGaussian++ with existing state-of-the-art approaches on the KITTI-360 dataset.

Methods	PSNR $\uparrow$	SSIM $\uparrow$
NeRF [7]	21.94	0.781
Point-NeRF [56]	21.54	0.793
NSG [12]	22.89	0.836
Mip-NeRF360 [27]	23.27	0.836
SUDS [35]	23.30	0.844
DNMP [57]	23.41	0.846
Ours-S	25.18	0.862
Ours-L	25.62	0.868

Table 4. TABLE IV : Quantitative Comparison of DrivingGaussian++ with State-of-the-Art Image-Editing Approaches. We compare our method with existing state-of-the-art techniques on the nuScenes dataset across various editing tasks. The results show competitive performance, with improved quantitative metrics and enhanced editing quality.

Methods	Type	${\text{CLIP}}_{dir}\uparrow$
Paint-by-Example [59]	2D	0.0282
AnyDoor [60]	2D	0.0027
Ours	3D	0.0866

Table 5. TABLE V : Effect of different initialization methods on the Gaussian model. LiDAR-600K † \dagger denotes downsampling the original LiDAR data to a corresponding point cloud magnitude. LiDAR-1M ‡ \ddagger denotes denoising and removing outliers in LiDAR points, which is used in our method.

Methods	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Random-600K	22.18	0.653	0.424
Random-1M	22.23	0.653	0.421
SfM-600K	28.36	0.851	0.256
NeRF-1M	28.51	0.858	0.251
LiDAR-600K $†$	28.49	0.854	0.245
LiDAR-1M $‡$	28.74	0.865	0.237
LiDAR-2M	28.78	0.867	0.237

Table 6. TABLE VI : Effect of each module in our proposed method. IS3G is short for the Incremental Static 3D Gaussians module, and CDGG is short for the Composite Dynamic Gaussian Graph module.

Model	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
w/o IS3G	27.72	0.771	0.295
w/o CDGG	26.97	0.752	0.306
w/o $L_{TSSIM}$	27.88	0.783	0.280
w/o $L_{Robust}$	28.05	0.814	0.271
w/o $L_{LiDAR}$	28.45	0.854	0.248
Ours-S	28.36	0.851	0.256
Ours-L	28.74	0.865	0.237

Equations

$${\mathbf{x}}_{\mathbf{p}}^{\mathbf{q}}=\mathbf{K}[{\mathbf{R}}_{\mathbf{t}}^{\mathbf{i}}\cdot l_s+{\mathbf{T}}_{\mathbf{t}}^{\mathbf{i}}],$$

$$p_{b_0}(l|\mu ,\mathbf{\Sigma })=e^{-\frac{1}{2}(\mathbf{l}-\mu {)}^{\top }{\mathbf{\Sigma }}^{-1}(\mathbf{l}-\mu )},$$

$${\hat{P}}_{b+1}(G_s)=P_b(G_s)\bigcup (x_{b+1},y_{b+1},z_{b+1}),$$

$$\hat{C}(G_s)=\sum _{b=1}^N{\Gamma }_b{\alpha }_b{C}_b,{\Gamma }_b=\prod _{i=1}^{b-1}(1-{\alpha }_b),$$

$$\tilde{C}=\varsigma (G_s)\sum \omega (\hat{C}(G_s)|\mathbf{R},\mathbf{T}),$$

$$H=<O,G_d,M,P,A,T>,$$

$${\mathbf{m}}_{\mathbf{o}}^{-1}={\mathbf{R}}_o^{-1}{\mathbf{S}}_o^{-1},$$

$${\alpha }_{o,t}=\sum \frac{(p_t-b_o{)}^2\cdot \cot a_o}{\Vert (b_o|{\mathbf{R}}_{\mathbf{o}},{\mathbf{S}}_{\mathbf{o}})-\rho {\Vert }^2}{\alpha }_{p_0},$$

$$G_{comp}=\sum H<O,G_d,M,P,A,T>+G_s,$$

$$\tilde{\mathbf{\Sigma }}=\mathbf{JE}\mathbf{\Sigma }{\mathbf{E}}^{\top }{\mathbf{J}}^{\top },$$

$$L_{TSSIM}(\delta )=1-\frac{1}{Z}\sum _{z=1}^{Z}SSIM(\Psi (\hat{C}),\Psi (C)),$$

$$L_{Robust}(\delta )=\kappa (\Vert \hat{I}-I{\Vert }_2),$$

$$L_{LiDAR}(\delta )=\frac{1}{s}\sum \Vert P(G_{comp})-L_s{\Vert }^2,$$

$$D_{opt}(M_{edit},x,y)=Average(D_{ori}(M_{edit},y)),$$

$$G_{snow}=\{G_k\},$$

$$p_{k,t+1}$$

$$s_{i,x},s_{i,y}=Sobe{l}_x(D_i),Sobe{l}_y(D_i),$$

$$N_i=(\frac{s_{i,x}}{\sqrt{s_{i,x}^2+s_{i,y}^2}},\frac{s_{i,y}}{\sqrt{s_{i,x}^2+s_{i,y}^2}},\frac{1}{\sqrt{s_{i,x}^2+s_{i,y}^2}}),$$

$$P_{snow}=\bigcup _{i=1}^n[P_i-KNN(\bigcup _{j=1}^{i-1}{P}_j,KDT(P_i),{\text{1}}_{th})],$$

$$P_{t+1}$$

$$Trajj\_pred$$

$$M_{inpaint}=\{G_i\in G_l|d(p_i,G_{del})<d_{thr}\},$$

Full text

1 Introduction

With the rapid advances of autonomous driving technologies, the availability of driving data has introduced new opportunities to enhance autonomous driving simulation. These datasets encapsulate rich semantic and spatial information, significantly improving the performance of downstream perception tasks. Among these, 3D scene editing is crucial in enhancing the data of autonomous driving. It facilitates the generation of diverse driving scenarios, which is particularly valuable in synthesizing complex and rare cases critical for testing and validation. Using 3D scene editing, it becomes possible to simulate diverse real-world driving conditions, thereby enhancing the robustness and safety of autonomous driving systems.

The 3D scene editing task encompasses a variety of components, including style transfer, motion modification, weather simulation, and the addition or removal of objects. However, due to the inherent diversity and complexity of these tasks, existing 3D scene editing methods [1, 2, 3, 4, 5, 6] are often highly specialized, lacking a framework capable of comprehensively addressing multiple editing tasks. While the widely adopted strategy of leveraging 2D editing techniques achieves superior results, it requires repeated 2D editing during training to maintain multi-view consistency. This significantly increases computational costs, making it impractical for large-scale autonomous driving scenarios.

Representing and modeling large-scale dynamic scenes is the foundation for 3D scene editing and contributes to a series of autonomous driving tasks. Unfortunately, reconstructing complex 3D scenes from sparse vehicle-mounted sensor data is challenging, particularly when the ego vehicle is moving at high speeds. Furthermore, it becomes even more challenging in multi-camera settings due to their outward views, minimal overlaps, and variations in light from different directions. Complex geometry, diverse optical degradation, and spatiotemporal inconsistency also pose significant challenges in modeling such a 360-degree, large-scale driving scene.

Neural radiance fields [7] (NeRFs) have recently emerged as a promising method for modeling scenery at the object or room level. Some recent studies [8, 9, 10, 11] have extended NeRF to large-scale, unbounded static scenes and multiple dynamic objects [12, 13]. However, NeRF-based methods are computationally intensive and require densely overlapping views and consistent lighting. These issues affect their ability to construct driving scenes with outward multi-camera setups at high speeds. Furthermore, network capacity limitations make it challenging for them to model long-term, dynamic scenes with multiple objects, leading to visual artifacts and blurring. The 3D Gaussian Splatting (3DGS) [14] methods represent scenes with explicit 3D Gaussian representation and achieve state-of-the-art performance in novel view synthesis. However, the original 3DGS still encounters significant challenges in modeling large-scale dynamic driving scenes due to fixed Gaussians and constrained representation capacity.

In this paper, our method is based on a framework that represents the surrounding and dynamic driving scenes. The key idea is to hierarchically model the complex driving scene using sequential data from multiple sensors. We adopt Composite Gaussian Splatting to decompose the scene into static background and dynamic objects, reconstructing each part separately. Based on these, global rendering via Gaussian Splatting captures occlusion in the real world, encompassing static backgrounds and dynamic objects. Furthermore, we incorporate a LiDAR prior into the Gaussian representation, which enables the recovery of more precise geometry and maintains better multi-view consistency.

We also introduce a multitask editing framework for autonomous driving scenes. Within this framework, we meticulously implement three key tasks: texture modification, weather simulation, and object manipulation. To mitigate computational and temporal costs, we propose the post-editing strategy, which effectively decouples the reconstruction and editing processes, as illustrated in Figure 2. We compare our method with IN2N [1], IGS2GS [15], and ClimateNeRF [6], which demonstrate superior processing speed and competitive performance. This approach enables editing without additional training, significantly reducing the overall training overhead. DrivingGaussian++ can be seamlessly applied to any pre-reconstructed 3D scene. Specifically, we employ Composite Gaussian Splatting to construct explicit scene representations using Gaussian distributions. By identifying and editing these Gaussians at the 3D level, we circumvent the issue of multi-view inconsistency that often arises when 2D models are directly applied to 3D spaces. Furthermore, we enhance the editing results by integrating advanced image-processing tools, demonstrating the potential of cross-dimensional technological integration in achieving high-quality scene modifications.

Differences with preliminary results published in CVPR 2024. We have extended our work in several aspects: (i) We represent large-scale, dynamic driving scenes based on Composite Gaussian Splatting, which introduces two novel modules, including Incremental Static 3D Gaussians and Composite Dynamic Gaussian Graphs. The former reconstructs the static background incrementally, while the latter models multiple dynamic objects with a Gaussian graph. (ii) We construct a scene editing framework to edit the reconstructed scene in a training-free manner, covering multiple tasks including texture modification, weather simulation, and object manipulation. It contributes to generating novel and realistic simulation data. (iii) We facilitate dynamic editing of driving scenes, which predicts the motion trajectory of particles inserted into the scene. (iv) We construct a foreground asset bank with 3D generation and reconstruction and validate the quality of the data.

2 Related Work

2.1 3D Reconstruction

2.1.1 Neural Radiance Fields

Rapid progress in neural rendering for novel view synthesis has received significant attention. Neural Radiance Fields (NeRFs), which utilizes multi-layer perceptrons (MLPs) and differentiable volume rendering, can reconstruct 3D scenes and synthesize novel views from a set of 2D images and corresponding camera pose information.

NeRF for Bounded Scene. Typical NeRF models are limited to bounded scenes, requiring a consistent distance between the center object and the camera. NeRF models also struggle with scenes captured with slight overlaps and outward capture methods. Numerous advances have expanded the capabilities of NeRFs, leading to improvements in training speed [16, 17, 18], pose optimization [19, 20, 21], scene editing [22, 6], and dynamic scene representation [23, 24]. However, applying NeRF to large-scale unbounded scenes, such as autonomous driving scenarios, remains challenging.

NeRF for Unbounded Scenes. For large-scale unbounded scenes, numerous methods [8, 9, 11, 10, 25] have introduced refined versions of NeRF to model multi-scale urban-level static scenes. Inspired by the mipmapping approach to preventing aliasing, MIP-NeRF models [26, 27] have been developed to account for unbounded scenes. To enable high-fidelity rendering, Xu et al. [28] combine the compact multi-resolution ground feature planes with NeRF for large urban scenes. On the other hand, Guo et al. [29] propose a disentanglement approach that can model unbounded street views but ignores dynamic objects on the road. However, these methods model scenes under the assumption that the scene remains static and face challenges in effectively capturing dynamic elements.

Although most existing NeRF-based methods rely on accurate camera poses, several approaches have been developed [30, 31] to synthesize the view of dynamic monocular videos. However, these methods are limited to forward monocular views and are unable to effectively handle inputs from surrounding multi-camera setups. For dynamic urban scenes, Ost et al. [12, 13] extend NeRF to dynamic scenes with multiple objects using a scene graph, and Wu et al. [32, 33] propose instance-aware realistic simulators for monocular dynamic scenes. On the other hand, Xie et al. [34] improve the parameterization and camera poses of the surrounding views while using LiDAR as additional depth supervision. In addition, several methods [35, 36] decompose the scene into static background and dynamic objects and construct the scene with the help of LiDAR and 2D optical flow.

The quality of views synthesized by NeRF-based methods deteriorates in scenarios with multiple dynamic objects and variations and lighting variations, owing to their dependency on ray sampling. In addition, the utilization of LiDAR is confined to providing auxiliary depth supervision, and its potential benefits in reconstruction, such as delivering geometric priors, have not been explored.

To address these limitations, we utilize Composite Gaussian Splatting to model the unbounded dynamic scenes, where the static background is incrementally reconstructed as the ego vehicle moves, and multiple dynamic objects are modeled and integrated into the entire scene through Gaussian graphs. LiDAR is employed as the initialization for Gaussians, providing a more accurate geometric shape prior and a comprehensive scene description rather than solely serving as depth supervision for images.

2.1.2 3D Gaussian Splatting

The recent 3D Gaussian Splatting method [14] represents a static scene with numerous 3D Gaussians and achieves state-of-the-art results in novel view synthesis and training speeds. Compared with existing explicit scene representations (e.g., mesh, voxels), the 3DGS can model complex shapes with fewer parameters. Unlike implicit neural rendering, the 3DGS allows fast rendering and differentiable computation with splat-based rasterization.

Dynamic 3D Gaussian Splatting. While the original 3DGS is designed to represent static scenes, several methods have been developed for dynamic objects/scenes. Given a set of dynamic monocular images, Yang et al. [37] introduce a deformation network to model the motion of Gaussians. In addition, Wu et al. [38] connect adjacent Gaussians via a HexPlane, enabling real-time rendering. However, these two approaches are explicitly designed for monocular single-camera scenes focused on a central object. Luiten et al. [39] parameterize the entire scene using a set of dynamic Gaussians that evolve. However, it requires a camera array with dense multi-view images as inputs.

In real-world autonomous driving scenes, the high-speed movement of data collection platforms leads to extensive and complex background variations, often captured by sparse views (e.g., 2-4 views). Moreover, fast-moving dynamic objects with intense spatial changes and occlusion further complicate the situation. Collectively, these factors pose significant challenges for existing methods.

2.2 3D Scene Cotrollable Editing

Representing and modeling 3D scenes is fundamental for novel view synthesis. Neural Radiation Field (NeRF) and 3D Gaussian Splatting are two prominent methods for 3D scene reconstruction. NeRF implicitly encodes scene geometry and appearance within a multilayer perceptron (MLP), while 3D Gaussian Splatting explicitly represents the scene using 3D Gaussian ellipsoids. Despite the demonstrated reconstruction capabilities, editing these representations remains a significant challenge. Current approaches can be broadly classified into two categories: editing guided by diffusion models and editing based on 3D particle systems.

2.2.1 Editing Based on Diffusion Guidance

Diffusion models have gained much attention for their ability to enable text-driven image editing. Recent methods [1, 5, 3, 40, 41, 4] have extended this capability to 3D scenes using pre-trained diffusion models. These methods add noise to images rendered from 3D models. The noisy images, augmented with additional control conditions, are processed by a 2D diffusion model, which predicts the noise that represents the discrepancy between the target output and the input. This predicted noise is then used to compute the Score Distillation Sampling (SDS) loss, guiding the optimization of the 3D model. Despite their effectiveness, these approaches struggle with maintaining multi-view consistency and managing complex scene dynamics, especially in large-scale environments. Instruct-NeRF2NeRF (IN2N) [1] uses text commands to edit 3D models by transforming the 3D editing task into a 2D image editing problem.

However, due to the inability to ensure consistent editing across surrounding views, this approach suffers from instability, slow processing speeds, and noticeable artifacts, particularly in 360-degree scenes. ViCA-NeRF [5] adopts a similar strategy to IN2N, selecting a subset of reference images from the scene dataset and editing them, while blending the remaining images based on the projection results. Although this blending approach mitigates some issues, it does not resolve consistency problems and often results in blurry edits.

Recently, DreamEditor [40] converts NeRF representations into mesh surfaces and directly optimizes the mesh using SDS loss and DreamBooth. HiFA [41] improves multi-view consistency by dynamically adjusting the diffusion timestep and reducing the weight of the noise signal. Gaussian-based approaches [3, 4] extend these NeRF editing techniques to 3D Gaussian Splatting, leveraging depth models to estimate image depth as a geometric prior for the conditional diffusion model. Although these methods achieve more consistent 3D editing, they are still limited to texture modification due to the fixed nature of depth estimation. Additionally, most of them rely on static 2D and 3D masks to constrain editing regions, which are ineffective for dynamic 3D model training. These approaches have been primarily validated on object-centric datasets and remain unexplored in complex driving scenes.

In contrast, DrivingGaussian++ employs a training-free paradigm to effectively address the challenges of existing methods in dynamic driving scene editing, achieving exceptional editing consistency and visual quality.

2.2.2 Editing Based on 3D Particle Systems.

In addition to text-driven editing guided by diffusion models, numerous domain-specific editing methods operate without relying on additional target images [6, 2]. ClimateNeRF [6] simulates particle-based entities for various weather conditions, such as snow, fog, and floods, and integrates them into the original neural field to achieve realistic weather simulation. Similarly, GaussianEditor and Infusion [2] leverage 3D Gaussian Splatting for controlled editing of 3D scenes. GaussianEditor identifies editing regions by training Gaussians with semantic attributes, enabling precise insertion and deletion at the 3D level. Infusion employs a depth completion model to establish depth information, which serves as a control signal for Gaussian completion. These methods demonstrate higher editing efficiency and superior multi-view consistency compared to diffusion model-guided approaches. Inspired by the recent advances, DrivingGaussian++ adopts 3D particle-level editing and further extends it to multiple tasks, including texture, object, and weather editing. Through a training-free paradigm, our approach achieves explicit, controllable, and efficient editing for large-scale autonomous driving scenes.

3 Method

Our goal is to achieve training-free editing within 3D autonomous driving scenes. To address multiple editing tasks, we propose a controllable and efficient framework. First, we accurately reconstruct dynamic driving scenes employing Composite Gaussian Splatting. Next, we identify specific Gaussians within the scene for modification or generate new Gaussians to simulate specific physical entities. These targeted Gaussians are then integrated into the original scene, where we predict the future trajectories of the objects. Finally, we refine the results using image processing techniques to enhance realism. Using this framework, we develop detailed editing methodologies for three key tasks: texture modification, weather simulation, and object manipulation. Our method is described in Fig. 3

3.1 Composite Gaussian Splatting

3DGS performs well in static scenes, but has significant limitations in mixed scenes involving large-scale static backgrounds and multiple dynamic objects. As illustrated in Figure 4, our objective is to represent surrounding large-scale driving scenes with Composite Gaussian Splatting for unbounded static backgrounds and dynamic objects.

3.1.1 LiDAR Prior with surrounding views

The primitive 3DGS attempts to initialize Gaussians via structure-from-motion (SfM). However, unbounded urban scenes for autonomous driving contain many multi-scale backgrounds and foregrounds. Nevertheless, they are only glimpsed through exceedingly sparse views, resulting in erroneous and incomplete recovery of geometric structures.

To provide better initialization for Gaussians, we introduce the LiDAR prior to 3D Gaussian to obtain better geometries and maintain multi-camera consistency in surrounding view registration. At each timestep $t\in T$, given a set of multi-camera images $\{I_{t}^{i}|i=1\ldots N\}$ collected from the moving platform and multi-frame LiDAR sweeps ${L_{t}}$. We minimize multi-camera registration errors using LiDAR-image multi-modal data and obtain accurate point positions and geometric priors.

We first merge multiple frames of LiDAR sweeps to obtain the complete point cloud of the scene, denoted as $L$. We follow Colmap [42] and extract image features $X={x_{p}^{q}}$ from each image individually. Next, we project the LiDAR points onto the surrounding images. For each LiDAR point $l$, we transform its coordinates to the camera coordinate system and match it with the 2D-pixel of the camera image plane through projection:

[TABLE]

where $\mathbf{x_{p}^{q}}$ is the 2D pixel of the image, $\mathbf{I_{t}^{i}}$, $\mathbf{R_{t}^{i}}$ and $\mathbf{T_{t}^{i}}$ are orthogonal rotation matrices and translation vectors, respectively. In addition, $\mathbf{K}\in\mathbb{R}^{3\times 3}$ represents the known camera intrinsic parameters. Notably, points from LiDAR might be projected onto multiple pixels across multiple images. Therefore, we select the point with the shortest Euclidean distance to the image plane and retain it as the projected point, assigning color.

Similar to existing 3D reconstruction methods [43, 44], we extend the dense bundle adjustment (DBA) to a multi-camera setup and obtain the updated LiDAR points. Experimental results show that initializing with LiDAR prior to aligning with surrounding multi-camera aids in providing the Gaussian model with more precise geometry priors.

3.1.2 Incremental Static 3D Gaussians

The static backgrounds of driving scenes pose challenges for scene modeling and editing due to their large scale, long duration, and variations caused by ego vehicle movement with multi-camera transformation. As a vehicle moves, the static background frequently undergoes temporal shifts and dynamic changes. Due to the perspective principle, prematurely incorporating distant street scenes from time steps far away from the current can lead to scale confusion, resulting in unpleasant artifacts and blurring. To address this issue, we improve 3DGS by introducing Incremental Static 3D Gaussians, leveraging the perspective changes introduced by the vehicle’s movement and the temporal relationships between adjacent frames, as shown in Figure 5.

We uniformly divide the static scene into $N$ bins based on the depth range provided by the LiDAR prior (Section 3.1.1). These bins are arranged in chronological order, denoted as $\{\textrm{b}_{i}\}^{N}$, where each bin contains multi-camera images from one or more time steps. Neighboring bins have a small overlap region, which is used to align the static backgrounds of two bins. The latter bin is then incrementally fused into the Gaussian field of the previous bins. For the scene within the first bin, we initialize the Gaussian model using the LiDAR prior (similarly applicable to SfM points):

[TABLE]

where $\mathbf{l}\in\mathbb{R}^{3}$ is the position of the LiDAR prior; $\mathbf{\mu}$ is the mean of the LiDAR points; $\mathbf{\Sigma}\in\mathbb{R}^{3\times 3}$ is an anisotropic covariance matrix; and ⊤ is the transpose operator. We utilize the surrounding views within this bin segment as supervision to update the parameters of the Gaussian model, including position $P(x,y,z)$, covariance matrix $\mathbf{\Sigma}$, coefficients of spherical harmonics for view-dependent color $C(r,g,b)$, along with an opacity $\alpha$.

For the subsequent bins, we use the Gaussians from the previous bin as the position priors and align the adjacent bins based on their overlapping regions. The 3D center for each bin can be defined as:

[TABLE]

where $\hat{P}$ is the collection of 3D center for Gaussians $G_{s}$ of all currently visible regions, $(x_{b+1},y_{b+1},z_{b+1})$ is the Gaussians coordinate within the $b+1$ region. We incorporate scenes from the subsequent bins into the previously constructed Gaussians with multiple surrounding frames as supervision. The incremental static Gaussian model $G_{s}$ is defined by:

[TABLE]

where $C$ denotes the color corresponding to each Gaussian in a certain view, $\alpha$ is the opacity, and $\Gamma$ is the accumulated transmittance of the scene according to $\alpha$ in all bins. During this process, the overlapping regions between surrounding multi-camera images are used to form the Gaussian models’ implicit alignment jointly.

Note that during the incremental construction of static Gaussian models, there may be differences in sampling the same scene between the front and rear cameras. As such, we use a weighted averaging to reconstruct the scene’s colors as accurately as possible during the 3D Gaussian projection:

[TABLE]

where $\tilde{C}$ is the optimized pixel color, $\varsigma$ denotes the differential splatting, $\omega$ is the weight for different views, $[\mathbf{R},\mathbf{T}]$ is the view-matirx for aligning multi-camera views.

3.1.3 Composite Dynamic Gaussian Graph

The autonomous driving environment is highly complex, involving multiple dynamic objects and temporal changes. As shown in Figure 5, objects are often observed from limited views (e.g., 2-4 views) due to the egocentric movements of the vehicle and dynamic objects. In addition, fast moving objects also lead to significant appearance, making it challenging to represent them using fixed Gaussians.

To address the challenges, we introduce the Composite Dynamic Gaussian Graph, enabling the construction of multiple dynamic objects in long-term, large-scale driving scenes. We first decompose dynamic foreground objects from static backgrounds to build the dynamic Gaussian graph using bounding boxes provided by the datasets. Dynamic objects are identified by their object ID and the corresponding timestamps of appearance. Additionally, the Grounded SAM Models [45] are employed for precise pixel-wise extraction of dynamic objects based on the range of bounding boxes.

We construct a dynamic Gaussian graph using

[TABLE]

where each node stores an instance object $o\in O$, $g_{i}\in G_{d}$ denotes the corresponding dynamic Gaussians, and $m_{o}\in M$ is the transform matrix for each object. Here, $p_{o}(x_{t},y_{t},z_{t})\in P$ is the center coordinate of the bounding box, and $a_{o}=(\theta_{t},\phi_{t})\in A$ is the orientation of the bounding box at time step $t\in T$. We compute one Gaussian separately for each dynamic object. Using the transformation matrix $m_{o}$, we transform the coordinate system of the target object $o$ to the world coordinate where the static background resides:

[TABLE]

where $\mathbf{R}_{o}^{-1}$ and $\mathbf{S}_{o}^{-1}$ are the rotation and translation matrices corresponding to each object.

After optimizing all nodes in the dynamic Gaussian graph, we combine dynamic objects and static backgrounds using a Composite Gaussian Graph. Each node’s Gaussian distribution is concatenated into the static Gaussian field based on the bounding box position and orientation in chronological order. In cases of occlusion between multiple dynamic objects, we adjust the opacity based on the distance from the camera center: closer objects have higher opacity, following the principles of light propagation:

[TABLE]

where $\alpha_{o,t}$ is the adjusted opacity of Gaussians for object $o$ at time step $t$, $p_{t}=(x_{t},y_{t},z_{t})$ is the center of Gaussians for the object. $[\mathbf{R_{o},S_{o}}]$ denotes the object-to-world transform matrix, $\rho$ denotes the center of camera view, and $\alpha_{p_{0}}$ is the opacity of Gaussians.

The composite Gaussian field, including both static background and multiple dynamic objects, is formulated by:

[TABLE]

where $G_{s}$ is obtained in Section 3.1.2 through Incremental Static 3D Gaussians and $H$ denots the optimized dynamic Gaussian graph.

3D driving scene editing is based on the composite Gaussians of static background and dynamic objects, reconstructed by Composite Gaussian Splatting, and performs multiple editing tasks on them without extra training.

3.2 Global Rendering via Gaussian Splatting

We adopt the differentiable 3D Gaussian splatting renderer $\varsigma$ from [14] and project the global composite 3D Gaussian into 2D with the covariance matrix $\widetilde{\Sigma}$:

[TABLE]

where $\mathbf{J}$ is the Jacobian matrix of the perspective projection, and $\mathbf{E}$ denotes the world-to-camera matrix.

The composite Gaussian field projects the global 3D Gaussian onto multiple 2D planes and is supervised using surrounding views at each time step. In the global rendering process, Gaussians from the next time step are initially invisible to the current and subsequently incorporated with the supervision of the corresponding global images.

The loss function of our method consists of three parts. Similar to [14, 46], we introduce the Tile Structural Similarity (TSSIM) to Gaussian Splatting, which measures the similarity between the rendered tile and the corresponding ground truth:

[TABLE]

where we split the screen into $M$ tiles, $\delta$ is the training parameters of the Gaussians, $\Psi(\hat{C})$ denotes the rendered tile from Composite Gaussian Splatting, and $\Psi(C)$ denotes the paired ground-truth tile.

We also introduce a robust loss to reduce outliers in 3D Gaussians, which can be defined as:

[TABLE]

where $\kappa\in(0,1]$ is the shape parameter that controls the robustness of the loss, $I$ and $\hat{I}$ denote the ground truth and the synthesis image, respectively.

The LiDAR loss is further employed by supervising the expected Gaussians’ position from the LiDAR, obtaining better geometric structure and edge shapes:

[TABLE]

where $P(G_{comp})$ is the position of 3D Gaussians, and $L_{s}$ is the LiDAR point prior.

We optimize the Composite Gaussians by minimizing the sum of three losses in Eq 11-13. The proposed editing method leverages globally rendered images to identify editing targets and utilizes depth information derived from 3DGS as geometry priors, enabling effective and realistic multitask editing.

3.3 *Controllable Editing for Dynamic Driving

Scenes*

We tackle three key editing tasks for autonomous driving simulations: texture modification, weather simulation, and object manipulation. To support these diverse editing tasks, we have developed a framework that sequentially operates on the Gaussians of the reconstructed scene using 3D geometric priors, Large Language Models (LLMs) for dynamic predictions, and advanced editing techniques to ensure overall coherence and realism.

Texture Modification: This task involves applying patterns to the surfaces of 3D objects. In autonomous driving, texture modification extends beyond aesthetics to allow the addition of critical road features, such as cracks, manhole covers, and signage, which are crucial to building more robust testing environments. We show failure cases of object detection model [47] in Figure 7, highlighting the importance of editing simulation. Before editing, the perception model accurately identifies objects within the scene. However, after editing with DrivingGaussian++, challenging cases in the 3D scene become undetectable to the model, providing a more effective testing environment to assess the reliability and robustness of various components within autonomous driving systems.

Weather Simulation: This task focuses on integrating dynamic meteorological phenomena, such as precipitation, snowfall, and fog, into autonomous driving scenarios. Weather simulation is critical for replicating driving conditions in severe weather, demonstrating its importance in augmenting training datasets.

Object Manipulation: This task is divided into object deletion and insertion within the reconstructed scene. Object insertion is further categorized into static and dynamic types, with dynamic insertion adaptively predicting the object’s motion trajectory. These manipulations are essential for building robust autonomous driving simulation systems.

To enable multitask editing, we propose a framework that performs sequential operations on the Gaussians of the reconstructed scene without extra training. The process begins by identifying target Gaussians to edit using 3D geometric priors, followed by their integration into the scene. We employ large-language models (LLMs) to predict dynamic object trajectories and apply image-processing techniques to refine the results, ensuring coherence and realism. The editing pipeline is illustrated in Figure 6.

3.3.1 Initialization

In the proposed editing framework, we refer to those introduced to or removed from the original scene as target Gaussians, while those reconstructed from the initial scene are termed original Gaussians. The approach for determining target Gaussians depends on the specific editing task. For object removal, the target Gaussians, corresponding to the subset of original Gaussians marked for removal, are identified by refining the 3D bounding box provided by the dataset. Since the LiDAR prior is integrated during the reconstruction process, we can accurately locate their positions without additional alignment of the coordinate system. For other editing tasks, new Gaussians are generated as target Gaussians, designed with specific shapes and distributions to meet the requirements of each task.

Texture Modification. We enhance the surface texture of an object by introducing new flat Gaussians onto the surface of the designated editing region. The process begins by selecting a viewpoint and using a diffusion model or similar tools to edit the original image, generating a target image and a corresponding mask to guide the 3D editing. Specifically, we randomly select a viewpoint that provides clear visibility of the target region and render the image to be edited along with its associated depth map. Next, we define the 2D mask of the target region and apply the diffusion model or image processing software to modify the image in a 2D space, producing the target image.

Using the target image and the mask, we generate target Gaussians and assign appropriate attributes through inverse projection. As shown in Fig. 8, DrivingGaussian++ projects the edited content onto the corresponding position based on the rendered depth map and pixel-wise correspondence.

However, discrepancies may arise between the surface reconstructed by 3D Gaussian Splatting and the actual object’s surface. These discrepancies can lead to inconsistencies between the rendered depth and the real depth of the object, potentially causing the surface of the target Gaussians to appear uneven and unrealistic, thereby compromising the editing quality.

To address this issue, we perform equalization on the depth map. Specifically, we normalize the depth of the editing area to ensure a relatively uniform depth distribution along the horizontal axis, while preserving the depth distribution along the vertical axis:

[TABLE]

where $D_{ori}$, $D_{opt}$ separately represent the rendered depth before and after the depth equalization, $M_{edit}$ denotes the binary mask of the editing region and $x$, $y$ are the image coordinates. This approach yields a flat surface for the target Gaussians, significantly enhancing the visual quality and realism of the texture modification.

Weather Simulation. We simulate weather particles by incorporating Gaussians with specific physical properties into the current scene and achieve dynamic effects by adjusting the positions of these Gaussians at each time step. The first step in weather simulation is to design particles that align with the desired physical properties. We calculate the number of original Gaussians along with the range of their positions and introduce new Gaussians with specific shapes and colors in a particular distribution within the scene. Specifically, we use narrow, semi-transparent, white Gaussians to represent raindrops, irregular white ellipsoidal Gaussians to represent snowflakes, and Gaussians following a random distribution in the scene to represent fog. As an example, for snow simulation, we define the target Gaussians $G_{snow}$ by:

[TABLE]

where the $k_{th}$ Gaussian $G_{k}$ satisfies $p_{k}=\zeta*(p_{\max}-p_{\min})+p_{\min}$, $c_{k}=(1,1,1)+\epsilon$, $s_{k,y}=\min(\min(s_{k,x},s_{k,x}),0)+\varepsilon$, and $p_{k}$, $c_{k}$, $s_{k,y}$ separately denotes its 3D coordinate, color, and scale attribute.

Second, to realize dynamic weather effects that include the falling of raindrops, the drifting of snowflakes, and the spread of fog, we add a specific trajectory to weather Gaussians according to the current time step. We describe the trajectory of snowflakes using an example:

[TABLE]

where $p_{k,t}$ denotes the position of $k_{th}$ Gaussian in $G_{snow}$ at timestep $t$, and $trajj\_func$ is a function that calculates the relative movement between consecutive positions in the time sequence.

We also implement the 3D snow coverage effect, as shown in Figure 8. Specifically, we first calculate the normal map of training images based on Depth-Anything [48] and the Sobel filter [49] as:

[TABLE]

[TABLE]

where $D_{i}$ and $N_{i}$ denote the depth and normal map of the image $i$, while $s_{i,x}$ and $s_{i,y}$ are gradient magnitudes in the horizontal and vertical directions. Based on the normal map, a snow mask is added in the region with a large vertical $y$ component. Using a processed image with snow coverage and rendered depth, an inverse projection is taken to calculate the snow particle Gaussians’ 3D positions from this viewpoint. Finally, we combine the positions under different viewpoints to realize a consistent snow-covering effect between frames. To avoid inconsistencies arising from repeated calculations in overlapping areas between frames, we construct a KD-Tree, and prune nodes that are closer to each other:

[TABLE]

where $P_{snow}$ denotes positions of the target snow particle Gaussians, $P_{i}$ denotes positions calculated from viewpoints in the $i_{th}$ frame, $KDT$ refers to the constructed KD-Tree, and $KNN$ stands for K Nearest Neighbor (KNN) function, which takes three parameters as input: the search range, the KDTree of the search target, and the number of top k neighbors. We insert the final target snow particle Gaussians into the scene and realize the snow-covering effect.

Object Manipulation. Due to the distinct nature of the operations, object insertion and deletion differ in their implementation. For object removal, the target Gaussians correspond to the object to be deleted. First, we extract the object’s 3D bounding box matrix from the dataset annotations and crop the Gaussians within the bounding box. To address holes caused by insufficient reconstruction in the occluded areas, we further use a diffusion model to locally paint the rendered image (Section 3.3.3). For object insertion, we construct a 3D foreground bank containing objects reconstructed using 3D Gaussian Splatting, which can be directly utilized for insertion. The objects in the bank are acquired through 3DGS reconstruction of Blender models collected online and sparse reconstruction of vehicles from autonomous driving datasets. Additionally, the lighting of the foreground object can be adjusted using MCLight [50] to better match the current scene.

3.3.2 Gaussians Composition with Trajectory Prediction

After identifying the target Gaussians, we integrate them with the original scene. This process aligns both components within the same coordinate system to establish physically accurate occlusion relationships. Notably, the covariance matrices of the two Gaussian groups may interfere with each other during rasterization rendering, potentially leading to blurry results. Thus, we perform an additional forward process for the added Gaussians and store the covariance matrices of the transformed Gaussians. Finally, the combined scene is rendered for visualization.

For the object insertion task, to ensure that dynamic objects have reasonable and diverse motion trajectories, we utilize the powerful scene comprehension capability of the Large Language Models [51] to predict the future trajectories of the inserted objects as:

[TABLE]

where $P_{t}$ denotes the position of inserted object at timestep $t$ while $P_{0}$ is the initial position, $Trajj\_pred(t)$ denotes the relative positions at time step $t$ generated by LLM, $dir_{sky}$ is the sky direction and $des$ denotes the description of the expected trajectory. Specifically, we take the initial vehicle’s position, the sky direction, and the trajectory description as prompts, and generate a series of possible future trajectory sequences through GPT-4o [51].

3.3.3 Global Refinement with Differentiable Rendering

Leveraging the recent advances in diffusion models and 2D image processing, our approach integrates these techniques to enhance the results of object manipulation tasks. For object removal, we use diffusion models to locally inpaint the damaged regions of the rendered image. First, we delete the target Gaussians of the specified region based on 3D annotations. However, due to occlusions and limitations in the data acquisition viewpoint, surrounding areas of the deleted Gaussians often contain artifacts or holes with poor reconstruction quality. To address this issue, we use the K Nearest Neighbor lgorithm to identify a set of Gaussians requiring repair around the target region. We then perform a binarization rendering on these Gaussians to generate the corresponding inpainting masks:

[TABLE]

where $M_{inpaint}$ is a binary mask with the Gaussians to be inpainted set to 1, $G_{l}$ represents the remaining Gaussians after removal, while $G_{del}$ denotes the removed Gaussians, $p_{i}$ denotes the position of $G_{i}$, $d_{thr}$ is the distance threshold that determines which Gaussians should be inpainted. The nearest distance between $G_{i}$ and the Gaussians in $G_{del}$ is given by $d(p_{i},G_{del})$, which is computed as $d(p_{i},G_{del})=KNN(p_{i},KDT(P_{del})$. Subsequently, the images to be repaired, along with corresponding masks, are fed into the diffusion model as inputs. DrivingGaussian++ performs partial inpainting to restore the integrity and visual authenticity of the scene, achieving more realistic and seamless object removal.

For the object insertion task, when extracting data from autonomous driving datasets, we perform sparse reconstruction to generate the target Gaussians. The reconstructed vehicles lack shadow information, which leads to a levitation effect in the rendered images. To enhance the realism of object insertion without additional training, we adopt a shadow synthesis approach inspired by ARShadowGAN [52]. Specifically, we synthesize shadows for the inserted objects to eliminate the levitation effect, thereby improving the visual consistency and realism of the scene.

4 Experiments

4.1 Datasets

The nuScenes [54] dataset comprises 1000 driving scenes collected using multiple sensors (6 cameras, 1 LiDAR, etc.). The images are annotated with accurate 3D bounding boxes from 23 object classes. Our experiments utilize the keyframes of six challenging scenes with surrounding views, collected from 6 cameras and corresponding LiDAR sweeps (optional), as input. The KITTI-360 [55] dataset contains multiple sensors, corresponding to over 320k images and point clouds. Although the dataset provides stereo camera images, we use only a single camera to demonstrate that our method also performs well in monocular scenes.

4.2 Implementation Details

Our implementation is based on the 3DGS framework, with fine-tuned optimization parameters to fit the large-scale unbounded scenes. Instead of using SfM points or randomly initialized points as input, we employ the LiDAR prior mentioned in Section 3.1.1 for initializations. Considering the computational cost, we use a voxel grid filter for LiDAR points, reducing the scale without losing geometric features. We employ random initialization for dynamic objects with initial points set to 3000, since objects are relatively small in large-scale scenes. We increase the total training iterations to 50,000, set the threshold for densifying grad to 0.001, and reset the opacity interval to 900. The learning rate of Incremental Static 3D Gaussians remains the same as in the official setting, while the learning rate of the Composite Dynamic Gaussian Graph exponentially decays from 1.6e-3 to 1.6e-6. We assess our models using various metrics, including PSNR, SSIM, and LPIPS, and report the average results of all camera frames in the scenes. All experiments are carried out on an 8 RTX8000 with 384 GB memory.

4.3 Reconstruction Results and Comparisons

4.3.1 Comparisons of surrounding views on nuScenes.

We evaluate the proposed model against state-of-the-art approaches, including NeRF-based methods [16, 26, 27, 53, 34, 35, 36] and 3DGS-based schemes [14]. As shown in Table I, our method outperforms Instant-NGP [16], which employs a hash-based NeRF for novel view synthesis. While Mip-NeRF [26] and Mip-NeRF360 [27] are designed specifically for unbounded outdoor scenes, our method performs favorably in all metrics.

Urban-NeRF [53] uses depth cues from LiDAR in a NeRF model to reconstruct urban scenes. In contrast, we leverage LiDAR as a geometric prior in the proposed Gaussian models to achieve more effective large-scale scene reconstruction. Our method performs favorably against S-NeRF [34] and SUDS [35], where both decompose the scene into static background and dynamic objects and construct the scene with the help of LiDAR. Compared to EmerNeRF [36], which applies a spatial-temporal representation for dynamic driving scenes using flow fields, our method achieves state-of-the-art results in all metrics, eliminating the need to estimate scene flow. For Gaussian-based approaches, our method enhances the performance of our baseline method, 3DGS [14], on large-scale scenes in all metrics.

We show qualitative evaluation results on challenging nuScenes driving scenes. For multicamera surround view synthesis, as shown in Figure 9, our method enables the generation of photorealistic rendering images and ensures view consistency across multicameras. Meanwhile, EmerNeRF [36] and 3DGS [14] do not perform well in challenging regions, exhibiting undesirable visual artifacts such as ghosting, dynamic object disappearance, loss of plant texture details, lane markings, and blurring in distant scenes.

We next demonstrate the reconstruction results for dynamic temporal scenes. Our method accurately models dynamic objects within large-scale scenes, mitigating issues such as loss, ghosting, or blurring of these dynamic elements. The proposed model consistently constructs dynamic objects over time, despite their relatively high speed of movement. As depicted in Figure 9, other approaches [36, 14] are inadequate for rapidly moving dynamic objects.

4.3.2 Comparisons of mono-view on KITTI-360

To further validate the effectiveness of our method in the setting of a monocular driving scene, we conduct experiments with the KITTI-360 data set with comparisons to existing state-of-the-art approaches, including NeRF [7], Mip-NeRF360 [27], Point-NeRF [56], NSG [12], SUDS [35], and DNMP [57]. As shown in Table III, our method performs favorably in monocular driving scenes against other models.

4.4 Editing Results and Comparisons

We first demonstrate our editing results on the nuScenes dataset for multiple tasks. Compared with state-of-the-art 2D and 3D editing methods, our approach achieves superior visual realism and better quantitative consistency.

To support flexible editing on driving scenes, we additionally created a 3D Gaussian foreground bank containing specialized driving scene objects. This foreground bank is critically important for autonomous driving simulation and model validation.

4.4.1 Qualitative Results and Comparisons

We perform training-free editing of the reconstructed nuScenes data by DrivingGaussian++ in three domains: texture, weather, and object manipulation. The comprehensive results are shown in Fig 12, showcasing the ability of DrivingGaussian++ to perform various editing operations in dynamic driving scenarios.

For weather editing, we achieve realistic effects through particle-based simulation, as described in Section 3.3.1. For snow simulation particularly, we add snow particle Gaussians at each timestep and estimate surface normals to determine deposition locations. This produces realistic snow accumulation, as shown in Figure 13. For object manipulation, we first obtain actors using 3D reconstruction or 4D generation methods such as DreamGaussian4D [58]. This allows for the insertion of dynamic objects, including non-rigid bodies such as humans and animals. By adapting the deformation module to foreground contexts, we achieve flexible and diverse dynamic object integration. Additionally, we employ LLM-based trajectory prediction to obtain the trajectory of inserted objects. The results of dynamic objects insertion are shown in Figure 15.

Fig 14 provides a performance comparison with existing 3D editing approaches. While InstructNeRF2Nerf [1] and InstructGS2GS [15] employ diffusion models for iterative 3D scene editing across multiple tasks, they exhibit limitations in maintaining photorealism and view consistency. ClimateNeRF [6] specializes in particle-level weather editing through surface normal computations, but its application lacks generalizability to other editing tasks and remains constrained to static environments. Our method addresses these limitations while achieving high-quality results across all editing tasks.

4.4.2 Quantitative Results and Comparisons

To evaluate the consistency and realism of our editing approach, we compare DrivingGaussian++ with state-of-the-art 3D and 2D editing techniques.

For 3D scene editing, we compare with ClimateNeRF [6], IN2N [1], and IGS2GS [15] in terms of task diversity, processing time, and CLIP-direction similarity [63]. As shown in Table II, DrivingGaussian++ performs favorably against all other methods in terms of diversity, efficiency, and text-aligned consistency. In particular, the editing time of DrivingGaussian++ is typically within 3$\sim$10 minutes for scenes from the NuScenes dataset, significantly lower than that of other 3D editing models that require long training time.

To evaluate the performance of DrivingGaussian++ on single-view editing, we also compare it with 2D editing methods [60, 59, 61, 64, 63, 62, 66] across different tasks, as shown in Table IV. For texture modification and object insertion, we compare with inpainting methods [60, 59, 61]. While AnyDoor [60] and Paint-by-Example [59] utilize 2D images for conditional editing, they produce inconsistent perspective relationships and poor consistency with the condition image. SD-Inpainting [61] takes text prompts and 2D masks as input, but suffers from limited performance and controllability. For weather simulation, we evaluate text-guided editing methods [64, 63, 62]. Although FreePromptEditing [64], InstructPix2Pix [63], and InstructDiffusion [62] exhibit good text understanding, their results often lack physical plausibility—for instance, snow is rendered merely as a stylistic change rather than as accumulated precipitation. InstructDiffusion [62] editing results in these weather scenes are less realistic. For object removal, we evaluate inpainting and text-guided methods [61, 62, 66]. SD-Inpainting [61] and InstructDiffusion [62] leave residual artifacts, while LaMa [66] introduces visible inconsistencies in scene restoration.

We evaluate editing consistency using CLIP direction similarity metric for texture, weather editing and object insertion. For object removal, we evaluate the quality using LPIPS and FID (as shown in SPIn-NeRF [67]). DrivingGaussian++ achieves superior performance on all tasks.

4.4.3 3D Gaussian Foreground Bank for Driving Scenes

We construct a comprehensive 3D Gaussian foreground bank containing various traffic elements: vehicles, bicycles, motorcycles, pedestrians, animals, and static objects such as signs and traffic cones. Figure 16 shows our foreground bank and insertion results.

Online Model Reconstruction. We collect 3D models (pedestrians, vehicles, etc.) from online sources and Chatsim [50], then reconstruct them using 3DGS [14]. For each model, we render 360°views in Blender and perform 3DGS reconstruction with COLMAP [42]. We adjust lighting using environment maps extracted from nuScenes.

Sparse Reconstruction of vehicles from nuScenes. We efficiently sparse reconstruct vehicles from nuScenes using SplatterImage [68]. Each vehicle requires about 2$\sim$4 reference images for Gaussian reconstruction.

Image-based Object Generation. To expand our dataset, we generate 3D objects with image inputs. We first extract clean object images with SAM [69]. Subsequently, we create static and dynamic 3D models using dreamgaussian [70] and dreamgaussian4d [58] for few-shot 3D generation, enabling the creation of static and dynamic objects with high fidelity and efficiency.

4.5 Ablation Study

4.5.1 Initialization prior for Gaussians

Comparative experiments are conducted to analyze the effect of different priors and initialization methods on the Gaussian model. The original 3DGS provides two initialization modes: randomly generated points and SfM points computed by COLMAP [42]. We additionally offer two other approaches: point clouds from a pre-trained NeRF model and points generated with LiDAR prior.

Meanwhile, to analyze the effect of point cloud quantity, we down-sample the LiDAR to 600K and apply adaptive filtering (1M) to control the number of generated LiDAR points. We also set different maximum thresholds for randomly generated points (600K and 1M). Here, SfM-600K$\pm$ 20K represents the number of points computed by COLMAP, NeRF-1M$\pm$20K denotes the total points generated by the pre-trained NeRF model, and LiDAR-2M$\pm$20k refers to the original quantity of LiDAR points.

As shown in Table V, randomly generated points lead to the worst results as they lack any geometric prior. Initializing with SfM points also cannot adequately recover the scene’s precise geometries due to the sparse points and intolerable structural errors. Leveraging point clouds generated from a pre-trained NeRF model provides a relatively accurate geometric prior, but there are still noticeable outliers. For the model initialized with the LiDAR prior, although downsampling results in loss of geometric information in some local regions, it still retains relatively accurate structural priors, thus surpassing SfM (Figure 18). We note that the experimental results do not change linearly with increasing LiDAR point quantities. This can be attributed to that overly dense points store redundant features that interfere with the optimization of the Gaussian model.

4.5.2 Effectiveness of Model Component

We analyze the contribution of each module of the proposed model. As shown in Table VI and Figure 17, the Composite Dynamic Gaussian Graph module plays a crucial role in reconstructing dynamic driving scenes, while the Incremental Static 3D Gaussians module enables high-quality large-scale background reconstruction. These two novel modules significantly enhance the modeling quality of complex driving scenes. Regarding the proposed loss functions, the ablation results indicate that both $L_{TSSIM}$ and $L_{Robust}$ significantly improve the rendering quality, improve texture details and remove artifacts. In addition, $L_{LiDAR}$ from LiDAR prior helps Gaussians achieve better geometric priors. The experimental results also demonstrate that DrivingGaussian++ performs well even without the prior LiDAR, demonstrating strong robustness for various initialization methods.

5 Conclusion

We introduce DrivingGaussian++, a framework for reconstructing and editing large-scale dynamic autonomous driving scenes. Our approach progressively models the static background using incremental static 3D Gaussians and captures multiple moving objects through a composite dynamic Gaussian graph. By leveraging LiDAR priors, we achieve accurate geometric structures and robust multi-view consistency, significantly enhancing the quality of scene reconstruction. DrivingGaussian++ facilitates training-free editing for tasks such as texture modification, weather simulation, and object manipulation, enabling the generation of realistic and diverse driving scenes. Experimental results on datasets such as nuScenes and KITTI-360 demonstrate that our framework achieves state-of-the-art performance in both reconstruction and editing tasks, enabling high-quality surrounding view synthesis and dynamic scene editing.