Don't Yell at Your Robot: Physical Correction as the Collaborative Interface for Language Model Powered Robots

TL;DR

This paper introduces a novel human-robot collaboration method where physical corrections by humans are used to improve the robot's understanding and execution of tasks driven by large language models, enabling more intuitive and effective interaction.

Contribution

The paper presents a new physical correction interface for LLM-powered robots, allowing real-time intention re-estimation and improved future interactions through natural language updates.

Findings

Physical corrections effectively update robot intentions.

Hybrid real-simulation experiments demonstrate feasibility.

Enhanced collaboration through physical interaction.

Abstract

We present a novel approach for enhancing human-robot collaboration using physical interactions for real-time error correction of large language model (LLM) powered robots. Unlike other methods that rely on verbal or text commands, the robot leverages an LLM to proactively executes 6 DoF linear Dynamical System (DS) commands using a description of the scene in natural language. During motion, a human can provide physical corrections, used to re-estimate the desired intention, also parameterized by linear DS. This corrected DS can be converted to natural language and used as part of the prompt to improve future LLM interactions. We provide proof-of-concept result in a hybrid real+sim experiment, showcasing physical interaction as a new possibility for LLM powered human-robot interface.