Lumina-4DGS: Illumination-Robust Four-Dimensional Gaussian Splatting for Dynamic Scene Reconstruction

TL;DR

Lumina-4DGS improves dynamic scene reconstruction by handling lighting inconsistencies from multiple cameras, leading to more accurate and stable 4D results.

Contribution

A novel framework that decouples photometric variations and ensures geometric accuracy through exposure compensation and SSIM-Gated Optimization.

Findings

Lumina-4DGS achieves a PSNR of 31.12 dB on the Waymo Open Dataset with minimal geometric errors.

The method shows a 2.13 dB PSNR improvement on a self-collected dataset with severe illumination shifts.

SSIM-Gated Optimization prevents overfitting by prioritizing geometric reliability during training.

Abstract

High-fidelity 4D reconstruction of dynamic scenes is pivotal for immersive simulation yet remains challenging due to the photometric inconsistencies inherent in multi-view sensor arrays. Standard 3D Gaussian Splatting (3DGS) strictly adheres to the brightness constancy assumption, failing to distinguish between intrinsic scene radiance and transient brightness shifts caused by independent auto-exposure (AE), auto-white-balance (AWB), and non-linear ISP processing. This misalignment often forces the optimization process to compensate for spectral discrepancies through incorrect geometric deformation, resulting in severe temporal flickering and spatial floating artifacts. To address these limitations, we present Lumina-4DGS, a robust framework that harmonizes spatiotemporal geometry modeling with a hierarchical exposure compensation strategy. Our approach explicitly decouples photometric variations into two levels: a Global Exposure Affine Module that neutralizes sensor-specific AE/AWB fluctuations and a Multi-Scale Bilateral Grid that residually corrects spatially varying non-linearities, such as vignetting, using luminance-based guidance. Crucially, to prevent these powerful appearance modules from masking geometric flaws, we introduce a novel SSIM-Gated Optimization mechanism. This strategy dynamically gates the gradient flow to the exposure modules based on structural similarity. By ensuring that photometric enhancement is only activated when the underlying geometry is structurally reliable, we effectively prioritize geometric accuracy over photometric overfitting. Extensive experiments validate the quantitative superiority of Lumina-4DGS. On the Waymo Open Dataset, our method achieves a state-of-the-art Full Image PSNR of 31.12 dB while minimizing geometric errors to a Depth RMSE of 1.89 m and Chamfer Distance of 0.215 m. Furthermore, on our highly challenging self-collected surround-view dataset featuring severe unconstrained illumination shifts, Lumina-4DGS yields a significant 2.13 dB PSNR improvement over recent driving-scene baselines. These results confirm that our framework achieves photorealistic, exposure-invariant novel view synthesis while maintaining superior geometric consistency across heterogeneous camera inputs.

Figures

Full text

1. Introduction

High-fidelity 4D reconstruction of dynamic scenes is a cornerstone of next-generation applications in virtual reality (VR), immersive simulation, and autonomous driving [1,2,3,4,5,6,7]. To ensure safety and realism, these systems require not only photorealistic novel view synthesis but also precise 3D geometric modeling of complex, moving environments. While Neural Radiance Fields (NeRFs) [8,9,10,11] have set high standards for rendering quality, their prohibitive computational costs hinder real-time deployment. Recently, 3D Gaussian Splatting (3DGS) [12] has emerged as a paradigm shift, enabling real-time rendering and rapid training by representing scenes with explicit, anisotropic 3D Gaussians. Despite its efficiency, applying 3DGS to real-world driving datasets reveals critical limitations rooted in the complex interplay between photometric inconsistency and geometric fidelity.

A fundamental challenge in reconstructing outdoor driving scenes is the violation of the brightness constancy assumption inherent in standard reconstruction pipelines. In multi-camera sensor arrays, independent auto-exposure (AE) and auto-white-balance (AWB) mechanisms induce significant brightness shifts across viewpoints and timestamps [13,14,15,16]. Furthermore, non-linear ISP effects and spatially varying lighting (e.g., shadows, vignetting) introduce local inconsistencies. As noted in recent studies, standard 3DGS fails to distinguish these transient photometric shifts from intrinsic scene radiance [12,17]. Consequently, the optimization process is forced to “cheat”: it compensates for spectral discrepancies by deforming the underlying geometry or generating “floater” artifacts to match the varying input images [4,18]. This results in severe temporal flickering and, more critically, compromised geometric accuracy, which is unacceptable for downstream tasks like obstacle avoidance or path planning.

To mitigate these issues, prior works have introduced appearance embeddings [2,19,20] or global affine transformations [18] to model exposure changes. While effective for global shifts, these methods struggle to capture high-frequency, spatially variant discrepancies common in dynamic environments. More recent approaches utilize bilateral grids for pixel-wise adjustments [21,22]. However, standard bilateral grids are notoriously difficult to optimize and prone to overfitting, often converging to unstable solutions that disrupt the scene’s structural coherence. Simultaneously, 3DGS itself suffers from a lack of geometric constraints; the discrete and unordered nature of Gaussians often leads to surfaces that are “fuzzy” or poorly aligned with ground truth depth, as they rely solely on photometric loss for supervision.

In this paper, we propose Lumina-4DGS, a robust framework that harmonizes spatiotemporal geometry modeling with a unified, hierarchical exposure compensation strategy. We argue that effective reconstruction requires explicitly decoupling intrinsic scene color from sensor-specific variations, but this decoupling must be strictly constrained to prevent appearance models from eroding geometric integrity. Our method integrates two key innovations. First, we propose a Spatiotemporal Hybrid Exposure Model centered on a Multi-Scale Bilateral Grid [22]. Unlike standard grids that struggle with optimization instability, our multi-scale design bridges the gap between global appearance codes and pixel-wise transformations. By unifying these paradigms, it adaptively captures broad exposure shifts at coarse scales while residually correcting fine-grained non-linearities at fine scales, all while imposing temporal smoothness constraints to prevent flickering. Second, to address the trade-off between photometric correction and geometric stability, we introduce a novel SSIM-Gated Optimization Mechanism. This strategy dynamically gates the gradient flow to the exposure modules based on Structural Similarity, ensuring that photometric enhancement is only activated when the underlying geometry is structurally reliable.

In summary, our contributions are as follows:

• (i)We propose a unified spatiotemporal exposure framework that effectively integrates a Global Exposure Affine Module with the Multi-Scale Bilateral Grid. By enforcing temporal smoothness constraints while decoupling sensor-level shifts from local variations, our framework ensures flicker-free rendering and robust convergence across heterogeneous cameras.
• (ii)We introduce a geometry-aware SSIM-Gated Optimization strategy to address the geometric degradation caused by powerful appearance models. By dynamically regulating the multi-scale grid based on structural similarity, we mitigate the texture-geometry ambiguity, achieving high-fidelity reconstruction without improving photometric scores at the expense of geometric accuracy.
• (iii)We validate our approach through extensive benchmarking on both public datasets (Waymo [23]) and a challenging self-collected driving dataset, showcasing notable improvements in rendering realism, temporal stability, and quantitative geometric metrics compared to existing baselines.

The remainder of this paper is organized as follows: Section 2 reviews the related work in dynamic scene reconstruction and exposure compensation. Section 3 details the methodology of our proposed Lumina-4DGS framework. Section 4 presents the experimental setup, quantitative and qualitative evaluations, and ablation studies. Section 5 discusses the broader implications and limitations of our method, and finally, Section 6 concludes the paper.

2. Related Work

2.1. Dynamic Scene Modeling for Autonomous Driving

Reconstructing dynamic driving environments is critical for simulation and autonomous system validation. Early approaches [4,24] utilized Neural Radiance Fields (NeRF) to model static backgrounds and dynamic objects separately. For instance, MARS [24] employs a modular NeRF framework, while NeuRAD [4] integrates sensor-specific effects like rolling shutter to improve realism. However, NeRF-based methods suffer from slow training and rendering speeds.

Recently, 3D Gaussian Splatting (3DGS) [12] has revolutionized this field with real-time performance. Methods like Street Gaussian [25] and DrivingGaussian [6] leverage explicit 3D Gaussians to represent urban scenes, enabling efficient rendering of dynamic agents. OmniRe [1] further constructs hierarchical scene representations to unify static backgrounds and dynamic entities. Beyond general driving scenarios, recent works have extended 3DGS to specialized downstream tasks. For example, ParkGaussian [26] introduces a slot-aware strategy to enhance reconstruction for parking slot perception in GPS-denied environments, while other research [27] combines 3DGS with adversarial domain adaptation to enable monocular robot navigation via sim-to-real transfer. Despite these advancements in scene representation and task-specific applications, most existing pipelines assume consistent illumination across views. When applied to multi-camera setups in the wild, independent auto-exposure (AE) and auto-white-balance (AWB) mechanisms break this assumption, leading to severe flickering and geometric artifacts [14,19,28]. Our work builds upon these foundations but specifically addresses the photometric inconsistencies inherent in raw sensor data to achieve robust 4D reconstruction.

2.2. Photometric Inconsistency and Appearance Modeling

To handle varying illumination and transient discrepancies (e.g., shadows, exposure shifts), “appearance embeddings” were popularized by NeRF-W [19] and subsequently adopted in 3DGS frameworks like WildGaussians [2] and SWAG [29]. Building on this paradigm, recent advancements have introduced more specialized mechanisms to address complex lighting artifacts. RobustSplat++ [30] identifies that standard Gaussian densification can overfit to transient illumination, proposing a delayed growth strategy combined with robust appearance modeling to decouple structural geometry from lighting disturbances. Similarly, in the context of endoscopic reconstruction, Endo-4DGX [31] tackles extreme low-light and over-exposure conditions by incorporating illumination embeddings with region-aware spatial adjustment modules. While these approaches represent significant progress in handling global style changes or domain-specific exposure challenges, they typically rely on latent codes or specialized training schedules. Consequently, they often lack the direct granularity to explicitly model high-frequency, spatially varying discrepancies such as vignetting or local contrast shifts [32] inherent in large-scale driving datasets.

Bilateral Grids [18,22] have long been established as powerful tools for edge-aware image enhancement. In the realm of neural rendering, recent methodologies [8,18,33] have adopted these grids to model spatially varying photometric effects. However, standard bilateral grids are high-dimensional and notoriously prone to optimization instability or overfitting when lacking sufficient constraints [22]. Distinct from these prior approaches that often rely on the grid to model the full spectrum of appearance changes, we adopt a hierarchical strategy. We limit the bilateral grid to specific local non-linearities while offloading sensor-level shifts to a global affine module. This explicit decoupling addresses the convergence issues inherent in previous grid-based methods, ensuring robust performance even under heterogeneous camera setups.

2.3. Geometric Consistency and Surface Reconstruction

Accurate geometry is pivotal for downstream tasks like obstacle avoidance. However, standard 3DGS is prone to geometric degradation, often representing surfaces as “fuzzy” point clouds or creating floating artifacts to minimize photometric loss [34,35]. Recent efforts like SuGaR [34] and 2DGS [36] attempt to improve geometry by explicitly enforcing surface constraints or employing planar primitives. SuGaR introduces density regularization to extract meshes, while 2DGS flattens Gaussians into disks to resolve geometric ambiguities in ray intersection.

However, these methods predominantly focus on the geometric representation itself, often overlooking the critical impact of photometric inconsistency on geometric convergence. In dynamic scenes with fluctuating exposure, powerful appearance models can inadvertently “explain away” geometric errors—a phenomenon known as texture-geometry ambiguity. For instance, a shadow or exposure shift might be incorrectly modeled as a geometric deformation rather than a lighting change. Unlike prior works that treat geometry and appearance optimization in isolation, we introduce a Geometry-Aware Optimization strategy. By gating the gradient flow based on structural similarity (SSIM), we ensure that photometric enhancements are applied only when the underlying structure is reliable, effectively preventing appearance models from corrupting the scene geometry.

3. Methodology

We present Lumina-4DGS, a robust framework designed to achieve high-fidelity 4D reconstruction from heterogeneous camera inputs. Built upon the foundation of the Dynamic Gaussian Scene Graph [6], our approach addresses the critical limitation of standard scene graph representations: their inability to decouple intrinsic scene radiance from transient, sensor-specific photometric variations (e.g., auto-exposure and white balance shifts).

As illustrated in Figure 1, we adopt a composite Gaussian Scene Graph as the geometric backbone, decomposing the complex environment into Sky, Background, and Dynamic Object nodes. While this graph structure effectively handles scene dynamics, direct optimization against inconsistent observations leads to geometric artifacts. To overcome this, we augment the scene graph rendering pipeline with a Hierarchical Exposure Compensation strategy. This module explicitly models the image formation process by coupling a global sensor-level affine transformation with a local multi-scale bilateral grid. Furthermore, to ensure that these appearance enhancements do not compromise the structural integrity of the scene graph, we introduce a Geometry-Aware SSIM-Gated Optimization strategy, which selectively gates gradients based on geometric reliability.

The remainder of this section is organized as follows: Section 3.1 formulates the reconstruction problem. Section 3.2 details our underlying Dynamic Gaussian Scene Graph representation. Section 3.3 introduces the Hierarchical Exposure Compensation mechanism. Finally, Section 3.4 describes the SSIM-gated optimization strategy.

3.1. Preliminaries: 3D Gaussian Splatting

We represent the static scene as a set of 3D Gaussians $[eqn]$ . Each Gaussian is defined by a center $[eqn]$ , a covariance matrix $[eqn]$ , an opacity $[eqn]$ , and view-dependent color coefficients $[eqn]$ (Spherical Harmonics). To ensure $[eqn]$ remains positive semi-definite during optimization, it is decomposed into a rotation matrix $[eqn]$ (parameterized by a quaternion $[eqn]$ ) and a scaling matrix $[eqn]$ (parameterized by a vector $[eqn]$ ):

$$[eqn]$$

Given a camera with viewing transformation $[eqn]$ and projective Jacobian $[eqn]$ , the 3D covariance is projected onto the 2D image plane as $[eqn]$ :

$$[eqn]$$

The pixel color $[eqn]$ at pixel coordinate $[eqn]$ is computed via volume rendering ( $[eqn]$ -blending). We let $[eqn]$ be the set of sorted Gaussians overlapping the pixel. The rendered color is accumulated as:

$$[eqn]$$

where $[eqn]$ is the viewing direction and $[eqn]$ is the 2D alpha contribution evaluated at $[eqn]$ :

$$[eqn]$$

Here, $[eqn]$ represents the intrinsic scene radiance, which is ideally consistent across views. However, in multi-camera driving datasets, the observed ground truth images $[eqn]$ are not a direct reflection of this intrinsic radiance due to independent auto-exposure (AE) and auto-white-balance (AWB) mechanisms. We model the observed image as:

$$[eqn]$$

where $[eqn]$ represents a complex, non-linear transformation that varies across camera c and timestamp t. Standard 3DGS minimizes the photometric error between $[eqn]$ and $[eqn]$ directly, forcing the Gaussians to bake these transient sensor effects into the geometry, causing floating artifacts. Our goal is to model $[eqn]$ explicitly to recover a consistent geometry.

3.2. Dynamic Gaussian Scene Graph Construction

To scale 3DGS to large-scale, dynamic driving environments, we construct a composite Dynamic Gaussian Scene Graph $[eqn]$ . As illustrated in Figure 2, this graph explicitly disentangles the scene into three semantic node types—Sky, Background, and Dynamic Objects—allowing us to incorporate rigorous kinematic constraints and decouple object motion from the static environment.

The global scene $[eqn]$ at time t is composed of the union of these nodes:

$$[eqn]$$

where $[eqn]$ represents the static urban geometry, $[eqn]$ models the far-field environment, and $[eqn]$ denotes the set of visible dynamic agents at time t.

3.2.1. Graph Node Definitions

Sky Node ( $[eqn]$ ): We model the sky using a Far-Field Environment Map representation. To address the infinite depth of the sky, we initialize $[eqn]$ as a set of Gaussians distributed on a large bounding sphere with radius $[eqn]$ . These Gaussians are translation-invariant relative to the camera, with their appearance $[eqn]$ dependent solely on the viewing direction $[eqn]$ . This handles the high-dynamic-range background without introducing depth artifacts.Background Node ( $[eqn]$ ): The static urban environment (e.g., roads, buildings, vegetation) is represented by stationary 3D Gaussians in the world frame. Their parameters optimize the time-invariant geometry of the scene, providing a stable geometric backbone.Dynamic Node ( $[eqn]$ ): Moving agents (vehicles, pedestrians) are handled via object-centric graphs. Instead of modeling them in world space directly, we maintain a set of canonical Gaussians $[eqn]$ in a local coordinate system for each object k. This allows the model to share geometric features across timestamps.

3.2.2. Rigid and Deformable Object Modeling

To accurately render dynamic agents, we map the canonical Gaussians to world space using timestamp-specific transformations.

Rigid Motion for Vehicles. For rigid objects such as cars, we utilize the tracked 6-DoF pose $[eqn]$ derived from off-the-shelf trackers. We explicitly transform the canonical parameters into world space. The world-space mean $[eqn]$ and rotation quaternion $[eqn]$ for the ith Gaussian of object k are computed as:

$$[eqn]$$ $$[eqn]$$

where ⊗ denotes quaternion multiplication and $[eqn]$ is the quaternion representation of $[eqn]$ . This formulation ensures that multi-view consistency is enforced via the object’s kinematic trajectory.

Deformable Motion for VRUs. Vulnerable Road Users (VRUs) like pedestrians exhibit non-rigid articulation. To handle this, we extend the rigid formulation with a time-dependent deformation field $[eqn]$ . We predict coordinate offsets $[eqn]$ and covariance corrections in the canonical space:

$$[eqn]$$

The final world-space position is obtained by applying the rigid pose to the deformed Gaussian:

$$[eqn]$$

This hybrid approach effectively decouples global trajectory from local articulated dynamics.

3.2.3. Graph Composition and Rasterization

At each rendering step, the dynamic scene graph is traversed to generate a unified set of 3D Gaussians in the world coordinate system. We let $[eqn]$ denote the transformation operator mapping local node parameters to world space. The composite scene is constructed as:

$$[eqn]$$

The rasterizer $[eqn]$ aggregates these nodes to produce the canonical image:

$$[eqn]$$

Crucially, $[eqn]$ aims to represent the consistent scene radiance before sensor processing. By explicitly separating dynamics from the static background in the graph, we can enforce strict geometric consistency constraints during composition (e.g., preventing dynamic objects from penetrating the static ground plane).

3.3. Hierarchical Exposure Compensation

As formulated in Equation (6), the observed image $[eqn]$ is contaminated by sensor-specific photometric variations. Direct optimization against these inconsistent observations forces 3D Gaussians to “bake in” transient lighting effects, resulting in “floater” artifacts. To resolve this, we propose a Hierarchical Exposure Compensation mechanism that explicitly models the camera response function (CRF). We decompose the mapping $[eqn]$ into a physically motivated two-stage pipeline:

$$[eqn]$$

This hierarchical design ensures that high-frequency local corrections are only applied after global histogram alignment, preventing the powerful local model from overfitting to global shifts.

3.3.1. Level 1: Global Exposure Affine Module

The primary source of photometric inconsistency in driving scenarios is the automatic adjustment of ISO gain and shutter speed (AE). We model this as a global, channel-wise affine transformation. For each camera c at timestamp t, we optimize a learnable gain embedding $[eqn]$ and a bias embedding $[eqn]$ . The intermediate globally compensated image $[eqn]$ is computed as:

$$[eqn]$$

where ⊙ denotes the element-wise Hadamard product. Physical Constraints: Crucially, we apply the exponential function $[eqn]$ to the gain vector. This enforces a strict positivity constraint ( $[eqn]$ ), consistent with the physics of photon accumulation, ensuring the adjusted radiance remains valid. This module effectively neutralizes broad histogram shifts and white balance discrepancies.

3.3.2. Level 2: Multi-Scale Bilateral Grid

While the global affine module addresses sensor-level shifts, it remains insufficient for spatially heterogeneous artifacts, such as lens vignetting and local tone mapping inconsistencies. To rectify these pixel-wise residuals while preserving high-frequency geometry, we introduce a Multi-Scale Bilateral Grid (Figure 3). Unlike heavy convolutional networks that risk overfitting or blurring textures, the bilateral grid offers an edge-aware, computationally efficient solution for real-time rendering.

Bilateral Grid Parameterization. We parameterize the local photometric response as a learnable 3D tensor $[eqn]$ . The grid dimensions $[eqn]$ and $[eqn]$ correspond to the spatial and luminance resolutions, respectively. Each voxel stores a flattened $[eqn]$ affine transformation matrix, allowing the grid to model complex local color twists rather than simple scalar scaling.

Content-Adaptive Slicing. To enable edge-aware filtering, the correction for any given pixel $[eqn]$ is conditioned on both its spatial location and its photometric intensity. We first extract a monochromatic guidance map $[eqn]$ from the globally aligned image  $[eqn]$ :

$$[eqn]$$

This guidance map lifts the 2D pixel coordinates into a 3D query space $[eqn]$ , where $[eqn]$ are normalized spatial coordinates and $[eqn]$ . We then retrieve a pixel-specific affine matrix $[eqn]$ via a differentiable trilinear interpolation (slicing) operator  $[eqn]$ :

$$[eqn]$$

Multi-Scale Residual Fusion. Photometric inconsistencies often manifest at varying frequencies—vignetting is globally smooth, whereas tone-mapping artifacts can be sharp. To capture this spectrum, we employ a multi-scale hierarchy with K grid levels (typically $[eqn]$ , with resolutions $[eqn]$ and $[eqn]$ ). The final compensated image $[eqn]$ is synthesized by accumulating residual corrections:

$$[eqn]$$

Here, the affine matrix $[eqn]$ operates on the homogeneous representation of the pixel color. This residual formulation ensures that the grid focuses solely on local non-linear refinements, maintaining the structural fidelity of the original Gaussian rendering.

3.4. Optimization Strategy

The complete forward rendering and gated optimization pipeline of Lumina-4DGS, which explicitly integrates independent auto-exposure (AE) and auto-white-balance (AWB) compensation, is summarized in Algorithm 1. Algorithm 1 Forward Rendering and Gated Optimization in Lumina-4DGS

• Require:  3D Gaussians $[eqn]$ , Camera Pose $[eqn]$ , Time t, Image $[eqn]$
• Ensure:  Final Rendered Image $[eqn]$ , Updated Parameters
• 1:Initialization: Global AE/AWB parameters ( $[eqn]$ ), Bilateral Grid $[eqn]$ , Threshold $[eqn]$
• 2://—Stage 1: Forward Rendering (Pixel-wise integration)—
• 3:for each ray r traversing sorted Gaussians do
• 4:   Calculate canonical color via Spherical Harmonics: $[eqn]$
• 5:   // Step A: Global AWB and AE Compensation
• 6:   Apply channel-wise gain (AWB) and global bias (AE):
• 7:       $[eqn]$
• 8:   // Step B: Local Photometric Refinement
• 9:   Sample grid affine matrix $[eqn]$ for spatial position $[eqn]$ :
• 10:       $[eqn]$
• 11:   Apply local affine correction:
• 12:       $[eqn]$
• 13:   // Step C: Volumetric Alpha Blending
• 14:   Accumulate transmittance $[eqn]$ and final color:
• 15:       $[eqn]$
• 16:end for
• 17:Assemble pixels to form $[eqn]$
• 18://—Stage 2: Object-Aware SSIM-Gated Optimization—
• 19:Render raw geometry image $[eqn]$ without exposure compensation
• 20:Calculate object-level structural reliability: $[eqn]$
• 21:if ** $[eqn]$ ** then
• 22:    $[eqn]$    // Structure is reliable, unfreeze appearance
• 23:else
• 24:    $[eqn]$    // Structure is chaotic, freeze appearance
• 25:end if
• 26://—Stage 3: Gradient Modulation—
• 27:Compute total loss $[eqn]$ (see Equation (20))
• 28:Update Geometry: $[eqn]$
• 29:Update Exposure: $[eqn]$
• 30:**return ** $[eqn]$

3.4.1. Object-Aware SSIM-Gating and the Brightness Constancy Assumption

A fundamental challenge in joint geometry-appearance optimization for 4D reconstruction is the frequent violation of the brightness constancy assumption. In dynamic driving scenes captured with independent auto-exposure (AE), this photometric inconsistency often triggers visual overfitting. When optimized without strict constraints, powerful appearance models (like our Multi-Scale Bilateral Grid) tend to misinterpret inter-frame brightness shifts as geometric density. This texture-geometry ambiguity causes the model to hallucinate textures onto dilated, erroneous geometric boundaries (an artifact of LiDAR voxelization), leading to severe ghosting and floating artifacts around dynamic objects.

To address this and preserve structural integrity while relaxing the brightness constancy requirement, we propose an Object-Aware SSIM-Gated Optimization strategy. Instead of applying a global constraint, our gating mechanism is applied at the object level. Utilizing 2D masks, we calculate the Structural Similarity Index (SSIM) independently for dynamic foreground objects and the static background. The SSIM between the raw geometry rendering $[eqn]$ and the ground truth $[eqn]$ serves as a dynamic proxy for geometric boundary reliability. We introduce an object-specific gating mask $[eqn]$ :

$$[eqn]$$

where $[eqn]$ is a progressive confidence threshold. The appearance gradients are modulated by this mask: $[eqn]$ . This ensures that if the structural-photometric mismatch is too high, the appearance model is frozen to prevent deforming the 4D geometry to minimize RGB loss.

Justification of the Thresholding Strategy ( $[eqn]$ ): Rather than using a static empirical value, the threshold $[eqn]$ is linearly annealed from $[eqn]$ to $[eqn]$ over the first 10,000 iterations. This dynamic curriculum is deeply grounded in the densification behavior of 3D Gaussian Splatting:

• Lower Bound (0.2): During the earliest iterations, the initial geometry projected from sparse LiDAR is highly chaotic, yielding an SSIM noise floor near 0.2. This lower bound strictly prevents the bilateral grid from compensating for catastrophic geometric initialization.
• Upper Bound (0.7): An $[eqn]$ of 0.7 indicates that the macro-structures and coarse object boundaries have sufficiently aligned with true image edges, safely overcoming the LiDAR voxel dilation. Beyond this point, residual errors are primarily dominated by sensor-specific illumination shifts, making it the mathematically optimal time to fully unfreeze the appearance model.
• 10k-Iteration Annealing: This smoothly synchronizes the appearance unfreezing with the most active geometric densification phase (splitting and cloning) of the 3D Gaussians.

Comparison with Simpler Alternatives: While gradient clipping restricts update magnitude, it does not prevent optimizing in the wrong direction. Conversely, a confidence-weighted loss slows down the learning of both appearance and geometry. Our SSIM gating strictly decouples the two: it entirely masks the appearance gradients ( $[eqn]$ ) to prevent hallucinatio while maintaining full gradient flow for the 3D Gaussians to rapidly correct their geometry against true 2D object boundaries.

3.4.2. Spatiotemporal Smoothness Constraints

To mitigate temporal flickering caused by independent per-frame optimization, we enforce smoothness constraints on the exposure parameters. Since auto-exposure (AE) and auto-white-balance (AWB) typically evolve smoothly over time, abrupt changes are penalized. We formulate the temporal smoothness loss $[eqn]$ by decoupling it into global and local components:

$$[eqn]$$

where $[eqn]$ represent the affine gain and bias at time t, and $[eqn]$ represents the coefficients of the bilateral grid. The first term enforces global exposure continuity, while the second ensures that spatially varying corrections (e.g., vignetting patterns) remain stable.

3.4.3. Initialization and Staged Training Schedule

Given that high-dimensional bilateral grids are prone to overfitting during early optimization, we treat our training schedule as an explicit structural regularizer. The Global Exposure Module is initialized to an identity mapping ( $[eqn]$ ), and all voxels within the Multi-Scale Bilateral Grid output identity affine matrices ( $[eqn]$ ).

We employ a delayed activation strategy. For the first 6000 iterations, the bilateral grid is completely frozen. This forces the optimization to rely solely on the Global Module to absorb massive, image-wide sensor shifts. Once global photometry is stabilized at iteration 6000, the multi-scale grid is unfrozen to learn only minimal, localized residual corrections, empirically eliminating severe color flickering.

3.4.4. Total Objective and Evaluation Rationale

The final training objective combines the standard reconstruction loss with our temporal and structural regularization terms:

$$[eqn]$$

To prevent the appearance models from generating degenerate color twists, $[eqn]$ penalizes the magnitude of exposure adjustments, forcing them to remain close to the identity mapping:

$$[eqn]$$

where $[eqn]$ denotes the flattened $[eqn]$ affine matrix at voxel v and $[eqn]$ is the identity matrix.

By explicitly modeling the photometric transformation (as detailed in Algorithm 1), our framework ensures that the underlying 4D geometry remains stable even when the brightness constancy assumption fails. Photometric inconsistency in dynamic driving scenes often triggers visual overfitting, where the model generates floating artifacts to compensate for inter-frame brightness shifts. This mismatch ultimately manifests as a degradation in standard metrics such as PSNR, SSIM, and LPIPS. Our subsequent evaluation thus focuses heavily on these photometric parameters, alongside geometric metrics (Depth RMSE and Chamfer Distance), to strictly validate the efficacy of our exposure-decoupled rendering and ensure improvements are grounded in physical structural integrity.

4. Experiments

4.1. Experimental Setup

4.1.1. Datasets

To validate the robustness of Lumina-4DGS under heterogeneous illumination conditions, we conduct experiments on two distinct datasets:

• 1.Waymo Open Dataset [23]: We utilize the official Perception v1.2 release with scene flow labels. Specifically, we evaluate on 12 challenging sequences (sequences 000–005 and 010–015), featuring significant lighting variations under rainy and overcast conditions, totaling approximately 1200 frames.
• 2.Custom Surround-View Dataset: To evaluate performance in unconstrained in-the-wild scenarios, we collected data using a vehicle-mounted rig of 6 cameras. The configuration is heterogeneous: the front-view and rear-view sensors capture at $[eqn]$ , while the four side-view sensors (front-left/right, rear-left/right) capture at $[eqn]$ . All cameras operate with fully independent auto-exposure (AE) enabled. The dataset is characterized by rapid inter-frame brightness shifts and extreme dynamic range changes across the 360° field of view. To facilitate geometric evaluation, we utilize accumulated LiDAR point clouds as the absolute ground truth.

4.1.2. Evaluation Metrics

Photometric inconsistency in dynamic driving scenes often triggers visual overfitting, where the model generates floating artifacts (e.g., volumetric fog) to compensate for inter-frame brightness shifts. Such structural-photometric mismatches ultimately manifest as a degradation in standard fidelity metrics. Therefore, we focus on both photometric and geometric parameters to validate the efficacy of our exposure-decoupled rendering.

Photometric Metrics: We report PSNR (↑), SSIM (↑), and LPIPS (↓). These metrics quantify the suppression of noise and flickering caused by rapid auto-exposure (AE) shifts. All metrics are computed between the final compensated rendering $[eqn]$ and the ground truth sensor images at their native resolutions.Geometric Metrics (LiDAR-based): To strictly validate the physical correctness and eliminate the aforementioned floating artifacts, we utilize LiDAR point clouds as absolute ground truth:

• -Depth RMSE (↓): Measures the root mean square error between rendered depth $[eqn]$ and projected LiDAR depth $[eqn]$ :

$$[eqn]$$

where $[eqn]$ denotes the set of pixels with valid LiDAR readings.

• -Chamfer Distance (CD) (↓): To assess 3D structural consistency beyond the 2D plane, we calculate the CD between the reconstructed Gaussian cloud $[eqn]$ and the GT LiDAR cloud $[eqn]$ :

$$[eqn]$$

This metric explicitly penalizes geometric deformations that do not align with physical measurements.

4.1.3. Baselines

We compare our method against state-of-the-art view synthesis approaches:

• 3DGS [12]: The vanilla 3D Gaussian Splatting baseline.
• StreetGS [25]: A representative dynamic urban scene reconstruction method based on 3DGS.
• OmniRe [1]: A state-of-the-art framework that constructs hierarchical scene representations to unify static backgrounds and dynamic entities.
• Uni-BG [22]: The recent state-of-the-art method (Wang et al., 2025) that unifies appearance codes and bilateral grids for driving scenes, serving as the direct baseline for our exposure compensation module.

4.1.4. Implementation Details

We implement Lumina-4DGS using PyTorch 2.0.1. Following the staged training schedule described in Section 3.4.3, the bilateral grid remains frozen for the first 6000 iterations and is subsequently unfrozen. The Multi-Scale Bilateral Grid is configured with a spatial resolution of $[eqn]$ and a luminance resolution of 8. The SSIM gating threshold $[eqn]$ is linearly annealed from $[eqn]$ to $[eqn]$ during the first 10k iterations. We train for 30k iterations on a single NVIDIA RTX 4090 GPU.

4.2. Comparative Analysis

4.2.1. Quantitative Evaluation on Public Benchmark

Table 1 summarizes the quantitative results on the Waymo Open Dataset. We compare against standard 3DGS, dynamic reconstruction methods (StreetGS [25]), and the recent state-of-the-art frameworks OmniRe [1] and Uni-BG [22]. To highlight the efficacy of our exposure compensation in recovering fine-grained details, we report metrics for the Full Image as well as specific foreground categories: Human and Vehicle.

Our method, Lumina-4DGS, outperforms all baselines across both full-frame and category-specific metrics. Notably, while Uni-BG [22] also utilizes multi-scale bilateral grids for appearance modeling, our hierarchical approach achieves a superior Full Image PSNR of 31.12 dB.

The most significant advantage is observed in the Human and Vehicle categories. Dynamic objects are frequently captured under rapidly changing relative illumination as they move; our hierarchical module effectively stabilizes these fluctuations. For instance, on the Human category, our method reaches 28.15 dB, which is a significant improvement over OmniRe (24.10 dB) and Uni-BG (23.45 dB). This demonstrates that our hierarchical decoupling prevents the model from misinterpreting exposure shifts as geometric blur on dynamic subjects.

Furthermore, regarding geometric accuracy, high photometric scores do not always correlate with correct geometry. While Uni-BG and OmniRe achieve high full-image PSNR, they suffer from higher Depth RMSE (2.15 m and 2.05 m, respectively). Thanks to our SSIM-Gated Optimization, Lumina-4DGS achieves the lowest Depth RMSE (1.89 m) and Chamfer Distance (0.215 m), proving that our method improves visual quality through physically grounded exposure modeling rather than geometric deformation.

Qualitative Comparison on the Waymo Dataset. To explicitly verify the visual advancement of our proposed method on the public benchmark, we provide a qualitative comparison in Figure 4. As illustrated, state-of-the-art baselines like OmniRe and Uni-BG struggle to decouple illumination from geometry under varying exposure. This failure causes them to overfit to brightness changes, generating severe floating artifacts—clearly visible as noisy clutter in the depth maps—to minimize photometric error. In contrast, Lumina-4DGS leverages Object-Aware SSIM-gating to explicitly enforce structural integrity, successfully removing these artifacts and producing sharp, clean renderings with highly accurate depth boundaries.

4.2.2. Quantitative Evaluation on Our Self-Collected In-the-Wild Dataset

While public benchmarks provide a standardized training ground, they represent an idealized” autonomous driving scenario. To rigorously evaluate robustness in unconstrained real-world environments, we conduct experiments on our Self-Collected Surround-View Dataset.

Unlike curated public datasets, our proprietary data were captured using a commercial sensor suite without lab-grade synchronization, introducing two distinct challenges:

• Photometric Inconsistency: independent auto-exposure (AE) and auto-white-balance (AWB) cause drastic brightness shifts.
• LiDAR-Vision FoV Mismatch: our setup exhibits a significant Field-of-View (FoV) gap between the 360° cameras and the sparse LiDAR.

Quantitative Results. As shown in Table 2, this domain gap causes a “performance collapse” for state-of-the-art baselines. Even recent methods designed for complex driving scenes, such as OmniRe [1] and Uni-BG [22], suffer drastic drops in both photometric and geometric accuracy. This confirms that methods relying heavily on accurate geometric initialization or standard bilateral grids fail when LiDAR supervision is sparse and illumination fluctuates drastically.

In contrast, Lumina-4DGS demonstrates remarkable robustness. By explicitly decoupling exposure from geometry and employing SSIM-Gating to handle the FoV mismatch, we achieve a Full Image PSNR of 27.23 dB (+2.13 dB over Uni-BG). Most importantly, our method significantly improves geometric fidelity in these unconstrained environments, reducing the Chamfer Distance (CD) to 0.315 m, compared to 0.456 m for OmniRe. This proves that our hierarchical approach and gated optimization effectively translate from idealized benchmarks to unconstrained, real-world fleet data by prioritizing structural integrity.

4.2.3. Qualitative Comparison

Figure 5 provides a detailed visual analysis of reconstruction quality under challenging independent auto-exposure conditions.

As evidenced in the second row, the baseline OmniRe exhibits severe geometry-texture ambiguity. In the absence of LiDAR supervision for the upper field of view (e.g., sky and distant buildings), the model misinterprets rapid photometric shifts as geometric density. This leads to the hallucination of floating artifacts—manifesting as volumetric fog or haze—and results in blurred, inconsistent textures.

In contrast, Lumina-4DGS (third row) employs our SSIM-Gated mechanism to explicitly disentangle sensor dynamics from scene geometry. By penalizing erroneous density growth in photometrically unstable regions, our method suppresses these artifacts, yielding clean, temporally stable renderings that preserve geometric integrity.

4.3. Geometric Consistency and Ablation Study

A key hypothesis of our work is that unconstrained exposure optimization leads to texture-geometry ambiguity, where the model generates geometric artifacts to explain away photometric differences. To verify the efficacy of our design choices and demonstrate how optimization values are affected at different levels of rendering, we conduct an ablation study on the Waymo Open Dataset by incrementally enabling the Global Exposure Module (Global), the Multi-Scale Bilateral Grid (Grid), and the Object-Aware SSIM-Gated Optimization (Gate).

Table 3 summarizes the quantitative results across these progressive rendering levels. We report both photometric metrics (PSNR, SSIM) and comprehensive geometric metrics, including Depth RMSE and Chamfer Distance (CD), to strictly evaluate structural fidelity.

Analysis of Results at Different Rendering Levels:

• Level 1 (Effect of Global Decoupling): Adding global compensation yields a significant boost in baseline photometric quality (PSNR increases from 28.15 dB to 30.50 dB). This confirms that large-scale, image-wide sensor shifts must be stabilized before localized refinement.
• Level 2 (The Overfitting Trap): When the Multi-Scale Bilateral Grid is introduced without gating, the model achieves peak photometric scores (PSNR: 32.78 dB). However, the geometric integrity degrades significantly, with Depth RMSE increasing to 2.95 m and Chamfer Distance rising to 0.438 m. This quantitatively proves that an unconstrained appearance model will visually overfit by deforming scene geometry to minimize RGB loss, manifesting as floating artifacts.
• Level 3 (Efficacy of SSIM-Gating): Enabling the Object-Aware SSIM-Gated Optimization successfully resolves this texture-geometry ambiguity. Although there is a minor, expected PSNR drop (−0.32 dB) due to restricted photometric freedom, the geometric accuracy improves drastically: RMSE drops to 1.89 m and CD reduces to an optimal 0.215 m. This demonstrates that our highest level of rendering effectively prioritizes correct physical structure over photometric overfitting.

Qualitative Evaluation of Ablation Modules. To explicitly verify the functions of our different modules and visually demonstrate the “overfitting trap,” we provide a qualitative ablation comparison in Figure 6. As rendering complexity increases to Level 2 (unconstrained Grid), the model forcibly alters the scene’s geometric density to fit complex lighting variations, resulting in noticeable depth degradation. Level 3 (Full Model) successfully cures this by gating the gradients, ensuring sharp boundaries and accurate depth reconstruction.

Analysis of Gating Dynamics

To explicitly demonstrate the operational mechanism of our Object-Aware SSIM-Gating, we visualize the optimization dynamics in Figure 7. During the early geometric warm-up phase (grey-shaded region), the object-level SSIM remains below the dynamically annealed threshold $[eqn]$ (linearly scaled from 0.2 to 0.7). Consequently, the appearance gradients (orange line) are strictly masked to zero.

This masking forces the 3D Gaussians to align with true 2D object boundaries solely via geometric adjustments, effectively correcting the coarse boundaries inherent to LiDAR voxelization. Once the structural fidelity exceeds the safety threshold $[eqn]$ (green-shaded region), the gate activates. This unfreezes the Bilateral Grid, allowing it to learn fine-grained photometric details while the underlying geometric integrity is securely preserved.

5. Discussion

The experimental results presented in Section 4 demonstrate that Lumina-4DGS effectively resolves the long-standing challenge of photorealistic reconstruction under unconstrained illumination conditions. In this section, we interpret these findings in the context of previous studies, analyze the underlying mechanisms of our success and discuss the broader implications for autonomous driving simulation.

5.1. Resolving the Texture-Geometry Ambiguity

A critical finding of our study is the confirmation of the “texture-geometry ambiguity” hypothesis. Previous state-of-the-art methods, such as OmniRe [1] and Street Gaussians [25], operate under the assumption that photometric consistency correlates directly with geometric accuracy. However, our ablation studies (Table 3) reveal that this assumption breaks down in “in-the-wild” scenarios with independent auto-exposure. When the rendering equation is forced to minimize the RGB loss against fluctuating brightness without explicit exposure decoupling, the optimizer resorts to “overfitting” by deforming the scene geometry, manifesting as the floating artifacts observed in Figure 5.

Our method fundamentally alters this optimization landscape. By introducing the Global Exposure Module, we mathematically disentangle sensor-induced sensitivity shifts from physical surface albedo. More importantly, SSIM-Gated Optimization serves as a structural regularizer. By rejecting gradient updates in regions where structural similarity is low (indicating a transient photometric error rather than a geometric misalignment), we force the Gaussian primitives to adhere to the physical scene geometry. This explains why Lumina-4DGS maintains the lowest Depth RMSE (1.89 m) even when photometrically outperforming baselines.

5.2. Bridging the Gap to Production Data

Most existing NeRF and 3DGS-based approaches are benchmarked on curated datasets like Waymo or NuScenes, which feature synchronized sensors and consistent exposure. Our experiments on the Self-Collected Surround-View Dataset highlight a significant “domain gap” between these idealized benchmarks and production-grade sensor data.

The performance collapse of baselines on our custom dataset (Table 2) underscores the fragility of current SOTA methods when facing LiDAR-Vision FoV mismatches and independent AE/AWB. Lumina-4DGS demonstrates that robust view synthesis in real-world applications requires modeling the sensor’s physical characteristics (e.g., ISO gain, vignetting) as part of the reconstruction pipeline. This capability is particularly valuable for building high-fidelity digital twins using low-cost, unsynchronized commercial fleets, significantly lowering the barrier to entry for large-scale data simulation.

5.3. Limitations and Future Directions

Despite these advancements, our current framework has limitations. First, while we effectively handle sensor-induced exposure changes, physical illumination changes caused by dynamic weather (e.g., moving cloud shadows, heavy rain, or snow) introduce complex light transport effects that our affine model cannot fully capture. Second, in regions with extreme motion blur or complete darkness, the SSIM gating mechanism may become overly conservative, potentially hindering geometry convergence.

Future research will focus on two directions: (1) integrating physics-based weather rendering models to separate environmental illumination from sensor exposure; and (2) extending our framework to an end-to-end neural sensor simulation pipeline, allowing for the synthesis of not just RGB images, but also raw sensor data with realistic noise profiles for downstream perception testing.

6. Conclusions

In this paper, we presented Lumina-4DGS, a novel framework designed to achieve illumination-robust 4D Gaussian Splatting for dynamic scene reconstruction. Addressing the limitations of existing methods in unconstrained environments, we identified the “texture-geometry ambiguity” as the primary obstacle where dynamic illumination shifts are often misinterpreted as geometric motion or structural noise.

To overcome this, we introduced a hierarchical exposure compensation pipeline integrated with an Object-Aware SSIM-Gated Optimization strategy. This approach effectively decouples sensor-induced photometric variations from the true temporal dynamics of the 4D scene. Extensive experiments on the Waymo Open Dataset and our challenging self-collected fleet dataset demonstrate that Lumina-4DGS not only achieves state-of-the-art rendering quality under rapid exposure changes but also recovers geometrically consistent structures, as validated against LiDAR ground truth.

By enabling robust reconstruction in the presence of independent auto-exposure and varying lighting conditions, Lumina-4DGS significantly closes the gap between idealized benchmarks and real-world autonomous driving data.

Moving forward, our future research will focus on two primary directions. First, leveraging our explicitly decoupled geometry and illumination representations, we plan to extend Lumina-4DGS to support controllable scene relighting and dynamic material editing. Second, we aim to investigate the framework’s robustness against more complex, physics-based environmental disturbances, such as severe weather conditions (e.g., rain, snow) and dynamic surface reflections. We believe these advancements will further solidify the foundation for creating high-fidelity, physically consistent digital twins for dynamic urban environments.