Play and Learn: Using Video Games to Train Computer Vision Models

TL;DR

This paper investigates whether synthetic RGB images from video games can effectively train computer vision models for tasks like segmentation and depth estimation, showing promising results comparable to real-world data.

Contribution

It demonstrates that synthetic data from video games can be used to train deep neural networks effectively, with simple domain adaptation improving real-world performance.

Findings

Synthetic images can improve model performance on real data.

Models trained on synthetic data achieve similar accuracy to those trained on real data.

Simple domain adaptation enhances the effectiveness of synthetic data.

Abstract

Video games are a compelling source of annotated data as they can readily provide fine-grained groundtruth for diverse tasks. However, it is not clear whether the synthetically generated data has enough resemblance to the real-world images to improve the performance of computer vision models in practice. We present experiments assessing the effectiveness on real-world data of systems trained on synthetic RGB images that are extracted from a video game. We collected over 60000 synthetic samples from a modern video game with similar conditions to the real-world CamVid and Cityscapes datasets. We provide several experiments to demonstrate that the synthetically generated RGB images can be used to improve the performance of deep neural networks on both image segmentation and depth estimation. These results show that a convolutional network trained on synthetic data achieves a similar test error to a network that is trained on real-world data for dense image classification. Furthermore, the synthetically generated RGB images can provide similar or better results compared to the real-world datasets if a simple domain adaptation technique is applied. Our results suggest that collaboration with game developers for an accessible interface to gather data is potentially a fruitful direction for future work in computer vision.