Robotic Programmer: Video Instructed Policy Code Generation for Robotic Manipulation

TL;DR

RoboPro is a foundation model enabling zero-shot robotic manipulation by synthesizing executable code from videos and instructions, achieving state-of-the-art success rates in simulation and real-world tasks.

Contribution

The paper introduces RoboPro, a robotic foundation model that perceives visual input and follows instructions to generate policy code in zero-shot settings, with a novel Video2Code synthesis method.

Findings

RoboPro surpasses previous models in zero-shot success rate by 11.6% on RLBench.

RoboPro performs comparably to supervised baselines in real-world tasks.

The model is robust to variations in API formats and skill sets.

Abstract

Zero-shot generalization across various robots, tasks and environments remains a significant challenge in robotic manipulation. Policy code generation methods use executable code to connect high-level task descriptions and low-level action sequences, leveraging the generalization capabilities of large language models and atomic skill libraries. In this work, we propose Robotic Programmer (RoboPro), a robotic foundation model, enabling the capability of perceiving visual information and following free-form instructions to perform robotic manipulation with policy code in a zero-shot manner. To address low efficiency and high cost in collecting runtime code data for robotic tasks, we devise Video2Code to synthesize executable code from extensive videos in-the-wild with off-the-shelf vision-language model and code-domain large language model. Extensive experiments show that RoboPro achieves the state-of-the-art zero-shot performance on robotic manipulation in both simulators and real-world environments. Specifically, the zero-shot success rate of RoboPro on RLBench surpasses the state-of-the-art model GPT-4o by 11.6%, which is even comparable to a strong supervised training baseline. Furthermore, RoboPro is robust to variations on API formats and skill sets.