Phone2Proc: Bringing Robust Robots Into Our Chaotic World

TL;DR

Phone2Proc enhances sim-to-real transfer for embodied agents by using quick phone scans and procedural scene generation, significantly improving real-world navigation success and robustness across diverse environments.

Contribution

It introduces a novel method combining phone scans and procedural generation to create realistic training environments for embodied AI agents.

Findings

Success rate in real-world ObjectNav increased from 34.7% to 70.7%.

Agents trained with Phone2Proc are robust to environmental changes.

Method requires only a 10-minute phone scan and a simple RGB camera.

Abstract

Training embodied agents in simulation has become mainstream for the embodied AI community. However, these agents often struggle when deployed in the physical world due to their inability to generalize to real-world environments. In this paper, we present Phone2Proc, a method that uses a 10-minute phone scan and conditional procedural generation to create a distribution of training scenes that are semantically similar to the target environment. The generated scenes are conditioned on the wall layout and arrangement of large objects from the scan, while also sampling lighting, clutter, surface textures, and instances of smaller objects with randomized placement and materials. Leveraging just a simple RGB camera, training with Phone2Proc shows massive improvements from 34.7% to 70.7% success rate in sim-to-real ObjectNav performance across a test suite of over 200 trials in diverse real-world environments, including homes, offices, and RoboTHOR. Furthermore, Phone2Proc's diverse distribution of generated scenes makes agents remarkably robust to changes in the real world, such as human movement, object rearrangement, lighting changes, or clutter.