Enabling robots to follow abstract instructions and complete complex dynamic tasks

TL;DR

This paper introduces a novel robotic framework that combines large language models, a curated knowledge base, and integrated feedback to enable robots to interpret high-level instructions and perform complex, dynamic tasks in unpredictable environments.

Contribution

The paper presents a new integrated approach using GPT-4, a curated database, and feedback mechanisms to improve robotic task execution from abstract instructions.

Findings

Successfully demonstrated in coffee making and plate decoration tasks

Enhanced robot adaptability to noise and disturbances during execution

Open-source implementation available for further research

Abstract

Completing complex tasks in unpredictable settings like home kitchens challenges robotic systems. These challenges include interpreting high-level human commands, such as "make me a hot beverage" and performing actions like pouring a precise amount of water into a moving mug. To address these challenges, we present a novel framework that combines Large Language Models (LLMs), a curated Knowledge Base, and Integrated Force and Visual Feedback (IFVF). Our approach interprets abstract instructions, performs long-horizon tasks, and handles various uncertainties. It utilises GPT-4 to analyse the user's query and surroundings, then generates code that accesses a curated database of functions during execution. It translates abstract instructions into actionable steps. Each step involves generating custom code by employing retrieval-augmented generalisation to pull IFVF-relevant examples from the Knowledge Base. IFVF allows the robot to respond to noise and disturbances during execution. We use coffee making and plate decoration to demonstrate our approach, including components ranging from pouring to drawer opening, each benefiting from distinct feedback types and methods. This novel advancement marks significant progress toward a scalable, efficient robotic framework for completing complex tasks in uncertain environments. Our findings are illustrated in an accompanying video and supported by an open-source GitHub repository (released upon paper acceptance).