Towards Efficient Architecture and Algorithms for Sensor Fusion

TL;DR

This paper introduces A3Fusion, a co-designed software-hardware system that enhances sensor fusion efficiency for automated vehicles by optimizing neural network architecture and deploying FPGA acceleration, balancing accuracy and latency.

Contribution

It presents an adaptive multi-modal learning network and a latency-aware optimization algorithm, along with an FPGA-based accelerator, for efficient middle sensor fusion in autonomous driving.

Findings

Achieves high semantic segmentation accuracy with reduced latency.

Demonstrates effective FPGA acceleration for middle fusion.

Balances accuracy and inference speed in sensor fusion.

Abstract

The safety of an automated vehicle hinges crucially upon the accuracy of perception and decision-making latency. Under these stringent requirements, future automated cars are usually equipped with multi-modal sensors such as cameras and LiDARs. The sensor fusion is adopted to provide a confident context of driving scenarios for better decision-making. A promising sensor fusion technique is middle fusion that combines the feature representations from intermediate layers that belong to different sensing modalities. However, achieving both the accuracy and latency efficiency is challenging for middle fusion, which is critical for driving automation applications. We present A3Fusion, a software-hardware system specialized for an adaptive, agile, and aligned fusion in driving automation. A3Fusion achieves a high efficiency for the middle fusion of multiple CNN-based modalities by proposing an adaptive multi-modal learning network architecture and a latency-aware, agile network architecture optimization algorithm that enhances semantic segmentation accuracy while taking the inference latency as a key trade-off. In addition, A3Fusion proposes a FPGA-based accelerator that captures unique data flow patterns of our middle fusion algorithm while reducing the overall compute overheads. We enable these contributions by co-designing the neural network, algorithm, and the accelerator architecture.