DEMONSTRATE: Zero-shot Language to Robotic Control via Multi-task Demonstration Learning

TL;DR

DEMONSTRATE introduces a novel approach that leverages task embeddings and inverse optimal control to enable zero-shot robotic control, reducing reliance on engineering-designed examples and improving robustness in manipulation tasks.

Contribution

The paper presents a new methodology that replaces in-context learning with embedding-based task representations and inverse optimal control, facilitating zero-shot control with minimal demonstrations.

Findings

Effective in simulation and hardware experiments

Reduces need for engineering-designed task examples

Enables assessment of hallucinations before execution

Abstract

The integration of large language models (LLMs) with control systems has demonstrated significant potential in various settings, such as task completion with a robotic manipulator. A main reason for this success is the ability of LLMs to perform in-context learning, which, however, strongly relies on the design of task examples, closely related to the target tasks. Consequently, employing LLMs to formulate optimal control problems often requires task examples that contain explicit mathematical expressions, designed by trained engineers. Furthermore, there is often no principled way to evaluate for hallucination before task execution. To address these challenges, we propose DEMONSTRATE, a novel methodology that avoids the use of LLMs for complex optimization problem generations, and instead only relies on the embedding representations of task descriptions. To do this, we leverage tools from inverse optimal control to replace in-context prompt examples with task demonstrations, as well as the concept of multitask learning, which ensures target and example task similarity by construction. Given the fact that hardware demonstrations can easily be collected using teleoperation or guidance of the robot, our approach significantly reduces the reliance on engineering expertise for designing in-context examples. Furthermore, the enforced multitask structure enables learning from few demonstrations and assessment of hallucinations prior to task execution. We demonstrate the effectiveness of our method through simulation and hardware experiments involving a robotic arm tasked with tabletop manipulation.