Memory-Efficient Dataflow Inference for Deep CNNs on FPGA

TL;DR

This paper introduces FCMP, a methodology to improve FPGA dataflow CNN accelerators by optimizing on-chip memory utilization, enabling cost-effective and flexible deployment of complex CNNs like ResNet-50 with minimal throughput loss.

Contribution

The paper presents FCMP, a novel memory packing technique that enhances OCM efficiency in FPGA CNN accelerators without hardware modifications, facilitating better porting and cost reduction.

Findings

Achieved up to 30% reduction in OCM utilization for CIFAR-10 accelerators.

Enabled porting of ResNet-50 accelerator from U250 to U280 with minimal throughput loss.

Demonstrated improved flexibility and cost-effectiveness in FPGA CNN inference designs.

Abstract

Custom dataflow Convolutional Neural Network (CNN) inference accelerators on FPGA are tailored to a specific CNN topology and store parameters in On-Chip Memory (OCM), resulting in high energy efficiency and low inference latency. However, in these accelerators the shapes of parameter memories are dictated by throughput constraints and do not map well to the underlying OCM, which becomes an implementation bottleneck. In this work, we propose an accelerator design methodology - Frequency Compensated Memory Packing (FCMP) - which improves the OCM utilization efficiency of dataflow accelerators with minimal reduction in throughput and no modifications to the physical structure of FPGA OCM. To validate our methodology, we apply it to several realizations of medium-sized CIFAR-10 inference accelerators and demonstrate up to 30% reduction in OCM utilization without loss of inference throughput, allowing us to port the accelerators from Xilinx Zynq 7020 to 7012S, reducing application cost. We also implement a custom dataflow FPGA inference accelerator for a quantized ResNet-50 CNN, utilizing on-chip weights, the largest topology ever implemented with this accelerator architecture. We demonstrate that by applying FCMP to the ResNet accelerator, the OCM bottleneck is alleviated which enables the accelerator to be ported from Alveo U250 to the smaller Alveo U280 board with less throughput loss compared to alternative techniques. By providing a finer-grained trade off between throughput and OCM requirements, FCMP increases the flexibility of custom dataflow CNN inference designs on FPGA.