IFTR: An Instance-Level Fusion Transformer for Visual Collaborative Perception

TL;DR

This paper introduces IFTR, a novel instance-level fusion transformer that enhances camera-only collaborative perception in autonomous driving by improving feature sharing, fusion, and communication efficiency, leading to significant performance gains.

Contribution

The paper proposes an instance-level fusion transformer (IFTR) that effectively fuses multi-agent visual features for collaborative perception, addressing the gap in camera-based systems and optimizing communication.

Findings

Achieved up to 57.96% improvement in AP@70 on DAIR-V2X dataset.

Demonstrated superior performance over state-of-the-art methods on multiple datasets.

Validated the effectiveness of key components through extensive experiments.

Abstract

Multi-agent collaborative perception has emerged as a widely recognized technology in the field of autonomous driving in recent years. However, current collaborative perception predominantly relies on LiDAR point clouds, with significantly less attention given to methods using camera images. This severely impedes the development of budget-constrained collaborative systems and the exploitation of the advantages offered by the camera modality. This work proposes an instance-level fusion transformer for visual collaborative perception (IFTR), which enhances the detection performance of camera-only collaborative perception systems through the communication and sharing of visual features. To capture the visual information from multiple agents, we design an instance feature aggregation that interacts with the visual features of individual agents using predefined grid-shaped bird eye view (BEV) queries, generating more comprehensive and accurate BEV features. Additionally, we devise a cross-domain query adaptation as a heuristic to fuse 2D priors, implicitly encoding the candidate positions of targets. Furthermore, IFTR optimizes communication efficiency by sending instance-level features, achieving an optimal performance-bandwidth trade-off. We evaluate the proposed IFTR on a real dataset, DAIR-V2X, and two simulated datasets, OPV2V and V2XSet, achieving performance improvements of 57.96%, 9.23% and 12.99% in AP@70 metrics compared to the previous SOTAs, respectively. Extensive experiments demonstrate the superiority of IFTR and the effectiveness of its key components. The code is available at https://github.com/wangsh0111/IFTR.