CAML: Collaborative Auxiliary Modality Learning for Multi-Agent Systems

TL;DR

CAML is a multi-agent framework that enables collaboration and data sharing during training, allowing effective inference with fewer modalities, improving decision-making and perception in autonomous systems.

Contribution

It introduces a novel multi-modal multi-agent learning framework that handles missing modalities during inference, addressing limitations of existing single-agent approaches.

Findings

Up to 58.1% improvement in accident detection for autonomous vehicles.

Up to 10.6% improvement in mIoU for collaborative semantic segmentation.

Effective multi-agent collaboration with reduced modalities during testing.

Abstract

Multi-modal learning has emerged as a key technique for improving performance across domains such as autonomous driving, robotics, and reasoning. However, in certain scenarios, particularly in resource-constrained environments, some modalities available during training may be absent during inference. While existing frameworks effectively utilize multiple data sources during training and enable inference with reduced modalities, they are primarily designed for single-agent settings. This poses a critical limitation in dynamic environments such as connected autonomous vehicles (CAV), where incomplete data coverage can lead to decision-making blind spots. Conversely, some works explore multi-agent collaboration but without addressing missing modality at test time. To overcome these limitations, we propose Collaborative Auxiliary Modality Learning (CAML), a novel multi-modal multi-agent framework that enables agents to collaborate and share multi-modal data during training, while allowing inference with reduced modalities during testing. Experimental results in collaborative decision-making for CAV in accident-prone scenarios demonstrate that CAML achieves up to a 58.1% improvement in accident detection. Additionally, we validate CAML on real-world aerial-ground robot data for collaborative semantic segmentation, achieving up to a 10.6% improvement in mIoU.