Evaluating Continual Learning Algorithms by Generating 3D Virtual Environments

TL;DR

This paper introduces a method to generate customizable, photo-realistic 3D virtual environments for evaluating continual learning algorithms, enabling more realistic and controlled experiments in vision tasks.

Contribution

It presents a novel parametric scene generator for 3D environments, accessible via a simple Python interface, to facilitate realistic continual learning research.

Findings

Generated scenes with controllable complexity and dynamics

Open-source tool available for the research community

Supports realistic, lifelong learning experiments in vision

Abstract

Continual learning refers to the ability of humans and animals to incrementally learn over time in a given environment. Trying to simulate this learning process in machines is a challenging task, also due to the inherent difficulty in creating conditions for designing continuously evolving dynamics that are typical of the real-world. Many existing research works usually involve training and testing of virtual agents on datasets of static images or short videos, considering sequences of distinct learning tasks. However, in order to devise continual learning algorithms that operate in more realistic conditions, it is fundamental to gain access to rich, fully customizable and controlled experimental playgrounds. Focussing on the specific case of vision, we thus propose to leverage recent advances in 3D virtual environments in order to approach the automatic generation of potentially life-long dynamic scenes with photo-realistic appearance. Scenes are composed of objects that move along variable routes with different and fully customizable timings, and randomness can also be included in their evolution. A novel element of this paper is that scenes are described in a parametric way, thus allowing the user to fully control the visual complexity of the input stream the agent perceives. These general principles are concretely implemented exploiting a recently published 3D virtual environment. The user can generate scenes without the need of having strong skills in computer graphics, since all the generation facilities are exposed through a simple high-level Python interface. We publicly share the proposed generator.