GauDP: Reinventing Multi-Agent Collaboration through Gaussian-Image Synergy in Diffusion Policies

TL;DR

GauDP introduces a Gaussian-image based shared scene representation that enhances scalable, perception-aware multi-agent collaboration by enabling agents to access task-critical features from a unified 3D Gaussian field.

Contribution

The paper proposes GauDP, a novel method for constructing and distributing a 3D Gaussian scene representation to improve multi-agent coordination without extra sensing modalities.

Findings

Outperforms existing image-based methods on RoboFactory benchmark

Approaches the effectiveness of point-cloud methods in multi-agent tasks

Maintains scalability with increasing number of agents

Abstract

Recently, effective coordination in embodied multi-agent systems has remained a fundamental challenge, particularly in scenarios where agents must balance individual perspectives with global environmental awareness. Existing approaches often struggle to balance fine-grained local control with comprehensive scene understanding, resulting in limited scalability and compromised collaboration quality. In this paper, we present GauDP, a novel Gaussian-image synergistic representation that facilitates scalable, perception-aware imitation learning in multi-agent collaborative systems. Specifically, GauDP constructs a globally consistent 3D Gaussian field from decentralized RGB observations, then dynamically redistributes 3D Gaussian attributes to each agent's local perspective. This enables all agents to adaptively query task-critical features from the shared scene representation while maintaining their individual viewpoints. This design facilitates both fine-grained control and globally coherent behavior without requiring additional sensing modalities (e.g., 3D point cloud). We evaluate GauDP on the RoboFactory benchmark, which includes diverse multi-arm manipulation tasks. Our method achieves superior performance over existing image-based methods and approaches the effectiveness of point-cloud-driven methods, while maintaining strong scalability as the number of agents increases.