Enabling Waypoint Generation for Collaborative Robots using LLMs and Mixed Reality

TL;DR

This paper introduces a framework combining large language models and augmented reality to simplify robot programming and deployment through natural language commands and visual feedback, demonstrated on a real pick-and-place task.

Contribution

It presents a novel integration of LLMs and AR for intuitive waypoint generation and robot programming, enabling non-expert users to deploy robots more easily.

Findings

Effective waypoint generation from natural language commands

Successful implementation on a real robot for pick-and-place tasks

Initial concept for expressive robot behavior and skill learning

Abstract

Programming a robotic is a complex task, as it demands the user to have a good command of specific programming languages and awareness of the robot's physical constraints. We propose a framework that simplifies robot deployment by allowing direct communication using natural language. It uses large language models (LLM) for prompt processing, workspace understanding, and waypoint generation. It also employs Augmented Reality (AR) to provide visual feedback of the planned outcome. We showcase the effectiveness of our framework with a simple pick-and-place task, which we implement on a real robot. Moreover, we present an early concept of expressive robot behavior and skill generation that can be used to communicate with the user and learn new skills (e.g., object grasping).