Tell Me Where to Go: A Composable Framework for Context-Aware Embodied Robot Navigation

TL;DR

This paper introduces NavCon, a framework that leverages Large Language Models to enable robots to interpret natural language commands for navigation by translating them into executable code, improving real-world contextual understanding.

Contribution

The paper presents NavCon, a novel intermediate layer that bridges LLMs and robot navigation, enhancing contextual command interpretation in diverse environments.

Findings

NavCon effectively interprets contextual commands across four environments.

The framework improves robot navigation accuracy with natural language instructions.

NavCon demonstrates adaptability to different command classes.

Abstract

Humans have the remarkable ability to navigate through unfamiliar environments by solely relying on our prior knowledge and descriptions of the environment. For robots to perform the same type of navigation, they need to be able to associate natural language descriptions with their associated physical environment with a limited amount of prior knowledge. Recently, Large Language Models (LLMs) have been able to reason over billions of parameters and utilize them in multi-modal chat-based natural language responses. However, LLMs lack real-world awareness and their outputs are not always predictable. In this work, we develop NavCon, a low-bandwidth framework that solves this lack of real-world generalization by creating an intermediate layer between an LLM and a robot navigation framework in the form of Python code. Our intermediate shoehorns the vast prior knowledge inherent in an LLM model into a series of input and output API instructions that a mobile robot can understand. We evaluate our method across four different environments and command classes on a mobile robot and highlight our NavCon's ability to interpret contextual commands.