Beyond Grand Theft Auto V for Training, Testing and Enhancing Deep Learning in Self Driving Cars

TL;DR

This paper explores using virtual environments like GTA V and Unity to generate labeled data for training and testing deep learning models in autonomous driving, aiming to improve safety and robustness.

Contribution

It introduces a large-scale virtual dataset from GTA V and the development of the Princeton Virtual Environment for training, testing, and enhancing self-driving AI models.

Findings

CNN trained on GTA V data effectively detects driving variables

Tested models perform well across different GTA V environments

PVE enables creation of rare corner cases for model improvement

Abstract

As an initial assessment, over 480,000 labeled virtual images of normal highway driving were readily generated in Grand Theft Auto V's virtual environment. Using these images, a CNN was trained to detect following distance to cars/objects ahead, lane markings, and driving angle (angular heading relative to lane centerline): all variables necessary for basic autonomous driving. Encouraging results were obtained when tested on over 50,000 labeled virtual images from substantially different GTA-V driving environments. This initial assessment begins to define both the range and scope of the labeled images needed for training as well as the range and scope of labeled images needed for testing the definition of boundaries and limitations of trained networks. It is the efficacy and flexibility of a "GTA-V"-like virtual environment that is expected to provide an efficient well-defined foundation for the training and testing of Convolutional Neural Networks for safe driving. Additionally, described is the Princeton Virtual Environment (PVE) for the training, testing and enhancement of safe driving AI, which is being developed using the video-game engine Unity. PVE is being developed to recreate rare but critical corner cases that can be used in re-training and enhancing machine learning models and understanding the limitations of current self driving models. The Florida Tesla crash is being used as an initial reference.