Using Natural Language for Human-Robot Collaboration in the Real World

TL;DR

This paper explores integrating large language models with physical robots to enhance natural language communication and collaboration with humans in real-world tasks, highlighting challenges and proof-of-concept experiments.

Contribution

It proposes a framework combining cognitive agents, LLMs, and robots, addressing key challenges in natural language understanding for human-robot collaboration.

Findings

Proof-of-concept experiments using ChatGPT demonstrate basic language understanding capabilities.

Discussion of challenges and potential solutions for real-world LLM-robot integration.

Insights into future development of operational language-capable robotic assistants.

Abstract

We have a vision of a day when autonomous robots can collaborate with humans as assistants in performing complex tasks in the physical world. This vision includes that the robots will have the ability to communicate with their human collaborators using language that is natural to the humans. Traditional Interactive Task Learning (ITL) systems have some of this ability, but the language they can understand is very limited. The advent of large language models (LLMs) provides an opportunity to greatly improve the language understanding of robots, yet integrating the language abilities of LLMs with robots that operate in the real physical world is a challenging problem. In this chapter we first review briefly a few commercial robot products that work closely with humans, and discuss how they could be much better collaborators with robust language abilities. We then explore how an AI system with a cognitive agent that controls a physical robot at its core, interacts with both a human and an LLM, and accumulates situational knowledge through its experiences, can be a possible approach to reach that vision. We focus on three specific challenges of having the robot understand natural language, and present a simple proof-of-concept experiment using ChatGPT for each. Finally, we discuss what it will take to turn these simple experiments into an operational system where LLM-assisted language understanding is a part of an integrated robotic assistant that uses language to collaborate with humans.