CORE: Cooperative Reconstruction for Multi-Agent Perception

TL;DR

CORE introduces a novel, communication-efficient multi-agent perception model that uses cooperative reconstruction to improve environment understanding, achieving state-of-the-art results in 3D detection and segmentation.

Contribution

The paper proposes a new cooperative reconstruction framework with efficient communication and supervision mechanisms for multi-agent perception tasks.

Findings

Achieves state-of-the-art performance on OPV2V dataset

Demonstrates improved perception accuracy with reduced communication costs

Effective in both 3D object detection and semantic segmentation

Abstract

This paper presents CORE, a conceptually simple, effective and communication-efficient model for multi-agent cooperative perception. It addresses the task from a novel perspective of cooperative reconstruction, based on two key insights: 1) cooperating agents together provide a more holistic observation of the environment, and 2) the holistic observation can serve as valuable supervision to explicitly guide the model learning how to reconstruct the ideal observation based on collaboration. CORE instantiates the idea with three major components: a compressor for each agent to create more compact feature representation for efficient broadcasting, a lightweight attentive collaboration component for cross-agent message aggregation, and a reconstruction module to reconstruct the observation based on aggregated feature representations. This learning-to-reconstruct idea is task-agnostic, and offers clear and reasonable supervision to inspire more effective collaboration, eventually promoting perception tasks. We validate CORE on OPV2V, a large-scale multi-agent percetion dataset, in two tasks, i.e., 3D object detection and semantic segmentation. Results demonstrate that the model achieves state-of-the-art performance on both tasks, and is more communication-efficient.