CoDeNet: Efficient Deployment of Input-Adaptive Object Detection on Embedded FPGAs

TL;DR

This paper presents CoDeNet, an FPGA-based object detection pipeline that efficiently integrates deformable convolutions, achieving high speed and accuracy with a tiny model size suitable for embedded systems.

Contribution

It introduces a novel FPGA-optimized object detection network with deformable convolutions, including quantization and design tradeoffs for improved efficiency and accuracy.

Findings

Achieves 26.9 FPS with 0.76 MB model size and 61.7 AP50 on Pascal VOC.

Attains 67.1 AP50 with 2.9 MB parameters, outperforming Tiny-YOLO.

Demonstrates effective FPGA deployment of input-adaptive object detection models.

Abstract

Deploying deep learning models on embedded systems has been challenging due to limited computing resources. The majority of existing work focuses on accelerating image classification, while other fundamental vision problems, such as object detection, have not been adequately addressed. Compared with image classification, detection problems are more sensitive to the spatial variance of objects, and therefore, require specialized convolutions to aggregate spatial information. To address this need, recent work introduces dynamic deformable convolution to augment regular convolutions. However, this will lead to inefficient memory accesses of inputs with existing hardware. In this work, we harness the flexibility of FPGAs to develop a novel object detection pipeline with deformable convolutions. We show the speed-accuracy tradeoffs for a set of algorithm modifications including irregular-access versus limited-range and fixed-shape. We then Co-Design a Network CoDeNet with the modified deformable convolution and quantize it to 4-bit weights and 8-bit activations. With our high-efficiency implementation, our solution reaches 26.9 frames per second with a tiny model size of 0.76 MB while achieving 61.7 AP50 on the standard object detection dataset, Pascal VOC. With our higher accuracy implementation, our model gets to 67.1 AP50 on Pascal VOC with only 2.9 MB of parameters-20.9x smaller but 10% more accurate than Tiny-YOLO.