Language-guided Robust Navigation for Mobile Robots in Dynamically-changing Environments

TL;DR

This paper presents a language-guided navigation system for mobile robots that detects environmental changes, queries humans for feedback, and integrates natural language instructions into global planning, validated through simulation and real-world tests.

Contribution

It introduces a novel method for monitoring robot plans, parsing natural language feedback into navigation waypoints, and robustly adapting to dynamic environments.

Findings

Effective detection of environmental changes impacting navigation

Successful integration of natural language feedback into planning

Validated robustness in simulation and real-world experiments

Abstract

In this paper, we develop an embodied AI system for human-in-the-loop navigation with a wheeled mobile robot. We propose a direct yet effective method of monitoring the robot's current plan to detect changes in the environment that impact the intended trajectory of the robot significantly and then query a human for feedback. We also develop a means to parse human feedback expressed in natural language into local navigation waypoints and integrate it into a global planning system, by leveraging a map of semantic features and an aligned obstacle map. Extensive testing in simulation and physical hardware experiments with a resource-constrained wheeled robot tasked to navigate in a real-world environment validate the efficacy and robustness of our method. This work can support applications like precision agriculture and construction, where persistent monitoring of the environment provides a human with information about the environment state.