TRoVE: Transforming Road Scene Datasets into Photorealistic Virtual Environments

TL;DR

This paper introduces TRoVE, a pipeline that transforms existing road scene datasets into photorealistic virtual environments, enhancing synthetic data quality and diversity for training intelligent vehicle systems.

Contribution

The work presents a novel synthetic data generation method that leverages existing datasets to produce high-fidelity, diverse virtual environments with minimal manual effort.

Findings

Improved mIoU scores in semantic segmentation tasks.

Enhanced data diversity and realism in synthetic environments.

Validated approach on Cityscapes and KITTI-STEP datasets.

Abstract

High-quality structured data with rich annotations are critical components in intelligent vehicle systems dealing with road scenes. However, data curation and annotation require intensive investments and yield low-diversity scenarios. The recently growing interest in synthetic data raises questions about the scope of improvement in such systems and the amount of manual work still required to produce high volumes and variations of simulated data. This work proposes a synthetic data generation pipeline that utilizes existing datasets, like nuScenes, to address the difficulties and domain-gaps present in simulated datasets. We show that using annotations and visual cues from existing datasets, we can facilitate automated multi-modal data generation, mimicking real scene properties with high-fidelity, along with mechanisms to diversify samples in a physically meaningful way. We demonstrate improvements in mIoU metrics by presenting qualitative and quantitative experiments with real and synthetic data for semantic segmentation on the Cityscapes and KITTI-STEP datasets. All relevant code and data is released on github (https://github.com/shubham1810/trove_toolkit).