Zero-shot Object-Centric Instruction Following: Integrating Foundation Models with Traditional Navigation

TL;DR

This paper introduces LIFGIF, a zero-shot method that combines foundation models with traditional navigation techniques to follow natural language instructions in novel environments, demonstrated through a new dataset and real-world robot tests.

Contribution

LIFGIF is the first approach to integrate foundation models with factor graph-based mapping for zero-shot object-centric instruction following in robotics.

Findings

LIFGIF outperforms state-of-the-art zero-shot baselines on OC-VLN dataset.

LIFGIF achieves robust navigation in novel environments.

Successful real-world deployment on Boston Dynamics Spot robot.

Abstract

Large scale scenes such as multifloor homes can be robustly and efficiently mapped with a 3D graph of landmarks estimated jointly with robot poses in a factor graph, a technique commonly used in commercial robots such as drones and robot vacuums. In this work, we propose Language-Inferred Factor Graph for Instruction Following (LIFGIF), a zero-shot method to ground natural language instructions in such a map. LIFGIF also includes a policy for following natural language navigation instructions in a novel environment while the map is constructed, enabling robust navigation performance in the physical world. To evaluate LIFGIF, we present a new dataset, Object-Centric VLN (OC-VLN), in order to evaluate grounding of object-centric natural language navigation instructions. We compare to two state-of-the-art zero-shot baselines from related tasks, Object Goal Navigation and Vision Language Navigation, to demonstrate that LIFGIF outperforms them across all our evaluation metrics on OCVLN. Finally, we successfully demonstrate the effectiveness of LIFGIF for performing zero-shot object-centric instruction following in the real world on a Boston Dynamics Spot robot.