READ: Large-Scale Neural Scene Rendering for Autonomous Driving

TL;DR

This paper introduces READ, a neural rendering method for synthesizing large-scale, photo-realistic autonomous driving scenes on a PC, enabling scene editing and stitching with improved coherence over traditional image-to-image translation.

Contribution

The paper proposes a novel neural rendering network that learns from sparse point clouds to synthesize and edit large-scale driving scenes, addressing limitations of existing methods.

Findings

Effective synthesis of large-scale driving scenes

Ability to stitch and edit scenes

Good performance demonstrated in experiments

Abstract

Synthesizing free-view photo-realistic images is an important task in multimedia. With the development of advanced driver assistance systems~(ADAS) and their applications in autonomous vehicles, experimenting with different scenarios becomes a challenge. Although the photo-realistic street scenes can be synthesized by image-to-image translation methods, which cannot produce coherent scenes due to the lack of 3D information. In this paper, a large-scale neural rendering method is proposed to synthesize the autonomous driving scene~(READ), which makes it possible to synthesize large-scale driving scenarios on a PC through a variety of sampling schemes. In order to represent driving scenarios, we propose an {\omega} rendering network to learn neural descriptors from sparse point clouds. Our model can not only synthesize realistic driving scenes but also stitch and edit driving scenes. Experiments show that our model performs well in large-scale driving scenarios.