G2D: from GTA to Data

TL;DR

G2D is a software tool that captures hyper-realistic, controllable videos from GTA V for computer vision research, enabling the creation of large-scale datasets with groundtruth camera poses for testing SfM algorithms.

Contribution

We introduce G2D, a novel tool that allows real-time manipulation of virtual environment conditions and automatic data capture from GTA V for computer vision applications.

Findings

G2D enables high-quality, controllable data collection from GTA V.

Generated datasets facilitate testing of structure-from-motion algorithms.

G2D is publicly available for research use.

Abstract

This document describes G2D, a software that enables capturing videos from Grand Theft Auto V (GTA V), a popular role playing game set in an expansive virtual city. The target users of our software are computer vision researchers who wish to collect hyper-realistic computer-generated imagery of a city from the street level, under controlled 6DOF camera poses and varying environmental conditions (weather, season, time of day, traffic density, etc.). G2D accesses/calls the native functions of the game; hence users can directly interact with G2D while playing the game. Specifically, G2D enables users to manipulate conditions of the virtual environment on the fly, while the gameplay camera is set to automatically retrace a predetermined 6DOF camera pose trajectory within the game coordinate system. Concurrently, automatic screen capture is executed while the virtual environment is being explored. G2D and its source code are publicly available at https://goo.gl/SS7fS6 In addition, we demonstrate an application of G2D to generate a large-scale dataset with groundtruth camera poses for testing structure-from-motion (SfM) algorithms. The dataset and generated 3D point clouds are also made available at https://goo.gl/DNzxHx