Self-Supervised Object Goal Navigation with In-Situ Finetuning

TL;DR

This paper introduces a self-supervised approach for object goal navigation that enables robots to learn and adapt in real-world environments without relying on expensive labeled 3D data, using location consistency as a key training signal.

Contribution

The work proposes a novel self-supervised training method called LocCon that allows in-situ finetuning of navigation models in real environments without labeled data.

Findings

LocCon outperforms models trained with 3D mesh annotations in real-world transfer.

Self-supervised in-situ training improves real-world navigation performance.

Models trained with LocCon are more robust and less affected by simulation artifacts.

Abstract

A household robot should be able to navigate to target objects without requiring users to first annotate everything in their home. Most current approaches to object navigation do not test on real robots and rely solely on reconstructed scans of houses and their expensively labeled semantic 3D meshes. In this work, our goal is to build an agent that builds self-supervised models of the world via exploration, the same as a child might - thus we (1) eschew the expense of labeled 3D mesh and (2) enable self-supervised in-situ finetuning in the real world. We identify a strong source of self-supervision (Location Consistency - LocCon) that can train all components of an ObjectNav agent, using unannotated simulated houses. Our key insight is that embodied agents can leverage location consistency as a self-supervision signal - collecting images from different views/angles and applying contrastive learning. We show that our agent can perform competitively in the real world and simulation. Our results also indicate that supervised training with 3D mesh annotations causes models to learn simulation artifacts, which are not transferrable to the real world. In contrast, our LocCon shows the most robust transfer in the real world among the set of models we compare to, and that the real-world performance of all models can be further improved with self-supervised LocCon in-situ training.