CoSense3D: an Agent-based Efficient Learning Framework for Collective Perception

TL;DR

CoSense3D introduces an agent-based training framework for collective perception that reduces resource demands and speeds up development without sacrificing model performance, facilitating safer and more efficient autonomous systems.

Contribution

This paper presents a novel agent-based training framework that separates data handling from deep learning modules, improving efficiency and flexibility in collective perception model development.

Findings

Significantly reduces GPU memory usage during training

Speeds up training time for collective perception models

Maintains high inference performance despite resource optimization

Abstract

Collective Perception has attracted significant attention in recent years due to its advantage for mitigating occlusion and expanding the field-of-view, thereby enhancing reliability, efficiency, and, most crucially, decision-making safety. However, developing collective perception models is highly resource demanding due to extensive requirements of processing input data for many agents, usually dozens of images and point clouds for a single frame. This not only slows down the model development process for collective perception but also impedes the utilization of larger models. In this paper, we propose an agent-based training framework that handles the deep learning modules and agent data separately to have a cleaner data flow structure. This framework not only provides an API for flexibly prototyping the data processing pipeline and defining the gradient calculation for each agent, but also provides the user interface for interactive training, testing and data visualization. Training experiment results of four collective object detection models on the prominent collective perception benchmark OPV2V show that the agent-based training can significantly reduce the GPU memory consumption and training time while retaining inference performance. The framework and model implementations are available at \url{https://github.com/YuanYunshuang/CoSense3D}