CoEnv: Driving Embodied Multi-Agent Collaboration via Compositional Environment

TL;DR

CoEnv introduces a novel framework combining real-world and simulation components to enhance multi-agent robotic collaboration through safe strategy exploration and reliable deployment.

Contribution

It presents the concept of compositional environment and a comprehensive CoEnv framework for improved multi-agent embodied manipulation.

Findings

High task success rates in multi-arm manipulation benchmarks

Effective safe sim-to-real transfer with collision detection

Enhanced coordination and planning in multi-agent systems

Abstract

Multi-agent embodied systems hold promise for complex collaborative manipulation, yet face critical challenges in spatial coordination, temporal reasoning, and shared workspace awareness. Inspired by human collaboration where cognitive planning occurs separately from physical execution, we introduce the concept of compositional environment -- a synergistic integration of real-world and simulation components that enables multiple robotic agents to perceive intentions and operate within a unified decision-making space. Building on this concept, we present CoEnv, a framework that leverages simulation for safe strategy exploration while ensuring reliable real-world deployment. CoEnv operates through three stages: real-to-sim scene reconstruction that digitizes physical workspaces, VLM-driven action synthesis supporting both real-time planning with high-level interfaces and iterative planning with code-based trajectory generation, and validated sim-to-real transfer with collision detection for safe deployment. Extensive experiments on challenging multi-arm manipulation benchmarks demonstrate CoEnv's effectiveness in achieving high task success rates and execution efficiency, establishing a new paradigm for multi-agent embodied AI.