RoboCoder: Robotic Learning from Basic Skills to General Tasks with Large Language Models

TL;DR

This paper introduces RoboCoder, a comprehensive benchmark and autonomous learning framework that enhances robotic generalization by leveraging large language models and real-time feedback to learn complex tasks from basic skills.

Contribution

It presents a new benchmark with diverse tasks and a novel adaptive learning framework that significantly improves robot generalization capabilities using LLMs.

Findings

GPT-4 achieved 47% pass rate in three-shot scenarios

RoboCoder framework improved performance by 36% relative

Benchmark includes 80 tasks across 7 entities

Abstract

The emergence of Large Language Models (LLMs) has improved the prospects for robotic tasks. However, existing benchmarks are still limited to single tasks with limited generalization capabilities. In this work, we introduce a comprehensive benchmark and an autonomous learning framework, RoboCoder aimed at enhancing the generalization capabilities of robots in complex environments. Unlike traditional methods that focus on single-task learning, our research emphasizes the development of a general-purpose robotic coding algorithm that enables robots to leverage basic skills to tackle increasingly complex tasks. The newly proposed benchmark consists of 80 manually designed tasks across 7 distinct entities, testing the models' ability to learn from minimal initial mastery. Initial testing revealed that even advanced models like GPT-4 could only achieve a 47% pass rate in three-shot scenarios with humanoid entities. To address these limitations, the RoboCoder framework integrates Large Language Models (LLMs) with a dynamic learning system that uses real-time environmental feedback to continuously update and refine action codes. This adaptive method showed a remarkable improvement, achieving a 36% relative improvement. Our codes will be released.