Hey Robot! Personalizing Robot Navigation through Model Predictive Control with a Large Language Model

TL;DR

This paper introduces a zero-shot approach that personalizes robot navigation by interpreting natural language instructions with a visual language model and adjusting a model predictive controller accordingly, enabling adaptable and safe robot behavior.

Contribution

The paper presents a novel zero-shot method that uses a visual language model to interpret user instructions and reconfigure robot navigation parameters in real-time, enhancing adaptability and safety.

Findings

Effective in simulation and real-world tests

Adapts to diverse environments and user preferences

Improves safety and user control in robot navigation

Abstract

Robot navigation methods allow mobile robots to operate in applications such as warehouses or hospitals. While the environment in which the robot operates imposes requirements on its navigation behavior, most existing methods do not allow the end-user to configure the robot's behavior and priorities, possibly leading to undesirable behavior (e.g., fast driving in a hospital). We propose a novel approach to adapt robot motion behavior based on natural language instructions provided by the end-user. Our zero-shot method uses an existing Visual Language Model to interpret a user text query or an image of the environment. This information is used to generate the cost function and reconfigure the parameters of a Model Predictive Controller, translating the user's instruction to the robot's motion behavior. This allows our method to safely and effectively navigate in dynamic and challenging environments. We extensively evaluate our method's individual components and demonstrate the effectiveness of our method on a ground robot in simulation and real-world experiments, and across a variety of environments and user specifications.