RoBridge: A Hierarchical Architecture Bridging Cognition and Execution for General Robotic Manipulation

TL;DR

RoBridge introduces a hierarchical architecture that combines cognitive reasoning and execution capabilities, significantly improving general robotic manipulation in open-ended scenarios through a novel integration of vision-language models and reinforcement learning.

Contribution

The paper presents RoBridge, a hierarchical system that bridges cognition and execution in robots, leveraging large-scale pre-trained models and reinforcement learning for enhanced generalization.

Findings

Achieves 75% success on new tasks

83% success rate in sim-to-real transfer with minimal data

Demonstrates effective integration of cognition and manipulation

Abstract

Operating robots in open-ended scenarios with diverse tasks is a crucial research and application direction in robotics. While recent progress in natural language processing and large multimodal models has enhanced robots' ability to understand complex instructions, robot manipulation still faces the procedural skill dilemma and the declarative skill dilemma in open environments. Existing methods often compromise cognitive and executive capabilities. To address these challenges, in this paper, we propose RoBridge, a hierarchical intelligent architecture for general robotic manipulation. It consists of a high-level cognitive planner (HCP) based on a large-scale pre-trained vision-language model (VLM), an invariant operable representation (IOR) serving as a symbolic bridge, and a generalist embodied agent (GEA). RoBridge maintains the declarative skill of VLM and unleashes the procedural skill of reinforcement learning, effectively bridging the gap between cognition and execution. RoBridge demonstrates significant performance improvements over existing baselines, achieving a 75% success rate on new tasks and an 83% average success rate in sim-to-real generalization using only five real-world data samples per task. This work represents a significant step towards integrating cognitive reasoning with physical execution in robotic systems, offering a new paradigm for general robotic manipulation.