LiFT: Lightweight, FPGA-tailored 3D object detection based on LiDAR data

TL;DR

LiFT is a lightweight, FPGA-optimized 3D object detection algorithm for LiDAR data that achieves high accuracy with minimal computational complexity, enabling real-time performance on resource-constrained platforms.

Contribution

The paper introduces LiFT, a novel FPGA-tailored 3D detection method that combines innovative architectures and quantization techniques to meet strict FPGA constraints.

Findings

Achieves 51.84% mAP on NuScenes validation dataset.

Uses only 20.73 GMACs, significantly below the 30 GMACs limit.

Outperforms comparable methods in accuracy and efficiency.

Abstract

This paper presents LiFT, a lightweight, fully quantized 3D object detection algorithm for LiDAR data, optimized for real-time inference on FPGA platforms. Through an in-depth analysis of FPGA-specific limitations, we identify a set of FPGA-induced constraints that shape the algorithm's design. These include a computational complexity limit of 30 GMACs (billion multiply-accumulate operations), INT8 quantization for weights and activations, 2D cell-based processing instead of 3D voxels, and minimal use of skip connections. To meet these constraints while maximizing performance, LiFT combines novel mechanisms with state-of-the-art techniques such as reparameterizable convolutions and fully sparse architecture. Key innovations include the Dual-bound Pillar Feature Net, which boosts performance without increasing complexity, and an efficient scheme for INT8 quantization of input features. With a computational cost of just 20.73 GMACs, LiFT stands out as one of the few algorithms targeting minimal-complexity 3D object detection. Among comparable methods, LiFT ranks first, achieving an mAP of 51.84% and an NDS of 61.01% on the challenging NuScenes validation dataset. The code will be available at https://github.com/vision-agh/lift.