An Extensible Framework for Open Heterogeneous Collaborative Perception

TL;DR

This paper introduces HEAL, an extensible framework for collaborative perception that effectively integrates new heterogeneous agents with minimal retraining, demonstrated on new datasets and outperforming state-of-the-art methods.

Contribution

Proposes HEAL, a novel framework enabling seamless integration of emerging heterogeneous agents into collaborative perception with low training costs.

Findings

HEAL outperforms SOTA methods on OPV2V-H and DAIR-V2X datasets.

HEAL reduces training parameters by 91.5% when adding new agent types.

Introduces OPV2V-H, a large-scale dataset with diverse sensor modalities.

Abstract

Collaborative perception aims to mitigate the limitations of single-agent perception, such as occlusions, by facilitating data exchange among multiple agents. However, most current works consider a homogeneous scenario where all agents use identity sensors and perception models. In reality, heterogeneous agent types may continually emerge and inevitably face a domain gap when collaborating with existing agents. In this paper, we introduce a new open heterogeneous problem: how to accommodate continually emerging new heterogeneous agent types into collaborative perception, while ensuring high perception performance and low integration cost? To address this problem, we propose HEterogeneous ALliance (HEAL), a novel extensible collaborative perception framework. HEAL first establishes a unified feature space with initial agents via a novel multi-scale foreground-aware Pyramid Fusion network. When heterogeneous new agents emerge with previously unseen modalities or models, we align them to the established unified space with an innovative backward alignment. This step only involves individual training on the new agent type, thus presenting extremely low training costs and high extensibility. To enrich agents' data heterogeneity, we bring OPV2V-H, a new large-scale dataset with more diverse sensor types. Extensive experiments on OPV2V-H and DAIR-V2X datasets show that HEAL surpasses SOTA methods in performance while reducing the training parameters by 91.5% when integrating 3 new agent types. We further implement a comprehensive codebase at: https://github.com/yifanlu0227/HEAL