DiffusionHarmonizer: Bridging Neural Reconstruction and Photorealistic Simulation with Online Diffusion Enhancer

TL;DR

DiffusionHarmonizer is an online generative framework that enhances neural scene renderings, reducing artifacts and improving realism for autonomous robot simulation, using a diffusion model adapted for real-time, scalable applications.

Contribution

It introduces a novel online enhancement method that transforms neural reconstructions into more realistic, artifact-free scenes with temporal consistency, suitable for real-time simulation.

Findings

Significantly reduces rendering artifacts in neural scene reconstructions.

Improves realism and temporal consistency of simulated scenes.

Operates efficiently on a single GPU in online environments.

Abstract

Simulation is essential to the development and evaluation of autonomous robots such as self-driving vehicles. Neural reconstruction is emerging as a promising solution as it enables simulating a wide variety of scenarios from real-world data alone in an automated and scalable way. However, while methods such as NeRF and 3D Gaussian Splatting can produce visually compelling results, they often exhibit artifacts particularly when rendering novel views, and fail to realistically integrate inserted dynamic objects, especially when they were captured from different scenes. To overcome these limitations, we introduce DiffusionHarmonizer, an online generative enhancement framework that transforms renderings from such imperfect scenes into temporally consistent outputs while improving their realism. At its core is a single-step temporally-conditioned enhancer that is converted from a pretrained multi-step image diffusion model, capable of running in online simulators on a single GPU. The key to training it effectively is a custom data curation pipeline that constructs synthetic-real pairs emphasizing appearance harmonization, artifact correction, and lighting realism. The result is a scalable system that significantly elevates simulation fidelity in both research and production environments.