Building Generalizable Agents with a Realistic and Rich 3D Environment

TL;DR

This paper introduces House3D, a comprehensive 3D environment with diverse, realistic scenes designed to improve the generalization of navigation agents through extensive data augmentation and multi-task training.

Contribution

The paper presents House3D, a large-scale, richly labeled 3D environment that enables advanced augmentation techniques to enhance agent generalization in unseen environments.

Findings

Agents with augmented training outperform baselines by over 8% in success rate

House3D facilitates scene-level and pixel-level augmentation strategies

Enhanced training leads to more robust navigation in unseen environments

Abstract

Teaching an agent to navigate in an unseen 3D environment is a challenging task, even in the event of simulated environments. To generalize to unseen environments, an agent needs to be robust to low-level variations (e.g. color, texture, object changes), and also high-level variations (e.g. layout changes of the environment). To improve overall generalization, all types of variations in the environment have to be taken under consideration via different level of data augmentation steps. To this end, we propose House3D, a rich, extensible and efficient environment that contains 45,622 human-designed 3D scenes of visually realistic houses, ranging from single-room studios to multi-storied houses, equipped with a diverse set of fully labeled 3D objects, textures and scene layouts, based on the SUNCG dataset (Song et.al.). The diversity in House3D opens the door towards scene-level augmentation, while the label-rich nature of House3D enables us to inject pixel- & task-level augmentations such as domain randomization (Toubin et. al.) and multi-task training. Using a subset of houses in House3D, we show that reinforcement learning agents trained with an enhancement of different levels of augmentations perform much better in unseen environments than our baselines with raw RGB input by over 8% in terms of navigation success rate. House3D is publicly available at http://github.com/facebookresearch/House3D.