Inference of Quantized Neural Networks on Heterogeneous All-Programmable Devices

TL;DR

This paper demonstrates real-time quantized neural network inference for object detection on an embedded all-programmable device, optimizing resource use across CPU and programmable logic to achieve 16 fps.

Contribution

It presents an end-to-end framework for quantized neural network inference on embedded hardware, integrating video processing and object detection.

Findings

Achieved 16 frames per second processing speed.

Implemented a fully integrated Darknet-based system.

Optimized resource utilization across CPU and programmable logic.

Abstract

Neural networks have established as a generic and powerful means to approach challenging problems such as image classification, object detection or decision making. Their successful employment foots on an enormous demand of compute. The quantization of network parameters and the processed data has proven a valuable measure to reduce the challenges of network inference so effectively that the feasible scope of applications is expanded even into the embedded domain. This paper describes the making of a real-time object detection in a live video stream processed on an embedded all-programmable device. The presented case illustrates how the required processing is tamed and parallelized across both the CPU cores and the programmable logic and how the most suitable resources and powerful extensions, such as NEON vectorization, are leveraged for the individual processing steps. The crafted result is an extended Darknet framework implementing a fully integrated, end-to-end solution from video capture over object annotation to video output applying neural network inference at different quantization levels running at 16~frames per second on an embedded Zynq UltraScale+ (XCZU3EG) platform.