Photorealism in Driving Simulations: Blending Generative Adversarial Image Synthesis with Rendering

TL;DR

This paper presents a hybrid neural graphics pipeline that enhances driving simulation realism by combining partial rendering of key objects with GAN-based synthesis of backgrounds, resulting in more photorealistic images for vehicle system testing.

Contribution

It introduces a novel approach that integrates partial rendering with GAN-based image synthesis to significantly improve visual fidelity in driving simulations.

Findings

Generated images are more photorealistic than traditional methods.

The approach outperforms conventional rendering in semantic retention and FID scores.

Images closely resemble real-world datasets like Cityscapes and KITTI.

Abstract

Driving simulators play a large role in developing and testing new intelligent vehicle systems. The visual fidelity of the simulation is critical for building vision-based algorithms and conducting human driver experiments. Low visual fidelity breaks immersion for human-in-the-loop driving experiments. Conventional computer graphics pipelines use detailed 3D models, meshes, textures, and rendering engines to generate 2D images from 3D scenes. These processes are labor-intensive, and they do not generate photorealistic imagery. Here we introduce a hybrid generative neural graphics pipeline for improving the visual fidelity of driving simulations. Given a 3D scene, we partially render only important objects of interest, such as vehicles, and use generative adversarial processes to synthesize the background and the rest of the image. To this end, we propose a novel image formation strategy to form 2D semantic images from 3D scenery consisting of simple object models without textures. These semantic images are then converted into photorealistic RGB images with a state-of-the-art Generative Adversarial Network (GAN) trained on real-world driving scenes. This replaces repetitiveness with randomly generated but photorealistic surfaces. Finally, the partially-rendered and GAN synthesized images are blended with a blending GAN. We show that the photorealism of images generated with the proposed method is more similar to real-world driving datasets such as Cityscapes and KITTI than conventional approaches. This comparison is made using semantic retention analysis and Frechet Inception Distance (FID) measurements.