Large Language Models for Orchestrating Bimanual Robots

TL;DR

This paper introduces LABOR, a novel approach using Large Language Models to coordinate bimanual robots for complex tasks, demonstrating improved success rates in simulation and providing insights into future research directions.

Contribution

The paper presents a new LLM-based framework for bimanual robot coordination, addressing the challenge of continuous space communication and demonstrating effectiveness in simulated long-horizon tasks.

Findings

Outperforms baseline in success rate on simulated tasks

Provides detailed analysis of failure cases and insights

Demonstrates potential of LLMs in complex robotic coordination

Abstract

Although there has been rapid progress in endowing robots with the ability to solve complex manipulation tasks, generating control policies for bimanual robots to solve tasks involving two hands is still challenging because of the difficulties in effective temporal and spatial coordination. With emergent abilities in terms of step-by-step reasoning and in-context learning, Large Language Models (LLMs) have demonstrated promising potential in a variety of robotic tasks. However, the nature of language communication via a single sequence of discrete symbols makes LLM-based coordination in continuous space a particular challenge for bimanual tasks. To tackle this challenge, we present LAnguage-model-based Bimanual ORchestration (LABOR), an agent utilizing an LLM to analyze task configurations and devise coordination control policies for addressing long-horizon bimanual tasks. We evaluate our method through simulated experiments involving two classes of long-horizon tasks using the NICOL humanoid robot. Our results demonstrate that our method outperforms the baseline in terms of success rate. Additionally, we thoroughly analyze failure cases, offering insights into LLM-based approaches in bimanual robotic control and revealing future research trends. The project website can be found at http://labor-agent.github.io.