CARLA2Real: a tool for reducing the sim2real appearance gap in CARLA simulator

TL;DR

This paper introduces CARLA2Real, a real-time tool that enhances the visual realism of CARLA simulator outputs to better match real-world datasets, thereby reducing the sim2real gap for autonomous driving research.

Contribution

The paper presents a novel, publicly available plugin for CARLA that improves photorealism in near real-time, aiding in more effective training of autonomous systems with synthetic data.

Findings

Enhanced synthetic data improves feature extraction and semantic segmentation.

The approach significantly reduces the visual gap between simulated and real data.

Comparison shows superiority over existing image-to-image translation methods.

Abstract

Simulators are indispensable for research in autonomous systems such as self-driving cars, autonomous robots, and drones. Despite significant progress in various simulation aspects, such as graphical realism, an evident gap persists between the virtual and real-world environments. Since the ultimate goal is to deploy the autonomous systems in the real world, reducing the sim2real gap is of utmost importance. In this paper, we employ a state-of-the-art approach to enhance the photorealism of simulated data, aligning them with the visual characteristics of real-world datasets. Based on this, we developed CARLA2Real, an easy-to-use, publicly available tool (plug-in) for the widely used and open-source CARLA simulator. This tool enhances the output of CARLA in near real-time, achieving a frame rate of 13 FPS, translating it to the visual style and realism of real-world datasets such as Cityscapes, KITTI, and Mapillary Vistas. By employing the proposed tool, we generated synthetic datasets from both the simulator and the enhancement model outputs, including their corresponding ground truth annotations for tasks related to autonomous driving. Then, we performed a number of experiments to evaluate the impact of the proposed approach on feature extraction and semantic segmentation methods when trained on the enhanced synthetic data. The results demonstrate that the sim2real appearance gap is significant and can indeed be reduced by the introduced approach. Comparisons with a state-of-the-art image-to-image translation approach are also provided. The tool, pre-trained models, and associated data for this work are available for download at: https://github.com/stefanos50/CARLA2Real.