RobustNav: Towards Benchmarking Robustness in Embodied Navigation

TL;DR

RobustNav introduces a benchmarking framework to evaluate embodied navigation agents' robustness against visual and dynamics corruptions, revealing significant performance drops and highlighting the need for further robustness research.

Contribution

The paper presents RobustNav, a novel benchmark for assessing robustness in embodied navigation, and provides systematic analysis of agent performance under various corruptions.

Findings

Standard agents underperform with corruptions

Data augmentation offers limited robustness gains

Significant performance gaps remain in corrupted settings

Abstract

As an attempt towards assessing the robustness of embodied navigation agents, we propose RobustNav, a framework to quantify the performance of embodied navigation agents when exposed to a wide variety of visual - affecting RGB inputs - and dynamics - affecting transition dynamics - corruptions. Most recent efforts in visual navigation have typically focused on generalizing to novel target environments with similar appearance and dynamics characteristics. With RobustNav, we find that some standard embodied navigation agents significantly underperform (or fail) in the presence of visual or dynamics corruptions. We systematically analyze the kind of idiosyncrasies that emerge in the behavior of such agents when operating under corruptions. Finally, for visual corruptions in RobustNav, we show that while standard techniques to improve robustness such as data-augmentation and self-supervised adaptation offer some zero-shot resistance and improvements in navigation performance, there is still a long way to go in terms of recovering lost performance relative to clean "non-corrupt" settings, warranting more research in this direction. Our code is available at https://github.com/allenai/robustnav