SAILenv: Learning in Virtual Visual Environments Made Simple

TL;DR

SAILenv is a user-friendly, customizable virtual 3D environment platform designed for visual recognition research, offering photorealistic scenes, pixel-level labeling, and motion data to facilitate algorithm testing.

Contribution

The paper introduces SAILenv, a novel, easy-to-use platform that provides photorealistic virtual environments with motion information, simplifying the testing of visual recognition algorithms.

Findings

Object detector trained on real images recognizes 3D objects in SAILenv.

Optical flow computation in SAILenv is efficient and GPU-compatible.

Platform enables realistic testing with minimal code and customization.

Abstract

Recently, researchers in Machine Learning algorithms, Computer Vision scientists, engineers and others, showed a growing interest in 3D simulators as a mean to artificially create experimental settings that are very close to those in the real world. However, most of the existing platforms to interface algorithms with 3D environments are often designed to setup navigation-related experiments, to study physical interactions, or to handle ad-hoc cases that are not thought to be customized, sometimes lacking a strong photorealistic appearance and an easy-to-use software interface. In this paper, we present a novel platform, SAILenv, that is specifically designed to be simple and customizable, and that allows researchers to experiment visual recognition in virtual 3D scenes. A few lines of code are needed to interface every algorithm with the virtual world, and non-3D-graphics experts can easily customize the 3D environment itself, exploiting a collection of photorealistic objects. Our framework yields pixel-level semantic and instance labeling, depth, and, to the best of our knowledge, it is the only one that provides motion-related information directly inherited from the 3D engine. The client-server communication operates at a low level, avoiding the overhead of HTTP-based data exchanges. We perform experiments using a state-of-the-art object detector trained on real-world images, showing that it is able to recognize the photorealistic 3D objects of our environment. The computational burden of the optical flow compares favourably with the estimation performed using modern GPU-based convolutional networks or more classic implementations. We believe that the scientific community will benefit from the easiness and high-quality of our framework to evaluate newly proposed algorithms in their own customized realistic conditions.