Instruction-Based Coordination of Heterogeneous Processing Units for Acceleration of DNN Inference

TL;DR

This paper introduces an instruction-based coordination architecture for FPGA systems with multiple PUs, enabling flexible DNN inference acceleration through programmable synchronization, dynamic deployment strategies, and efficient model partitioning.

Contribution

It proposes a novel instruction-based coordination architecture and a compilation framework for flexible, efficient DNN inference on FPGA with multiple PUs.

Findings

Achieves up to 98% compute efficiency.

Provides up to 2.7x throughput gains.

Supports dynamic runtime switching among deployment strategies.

Abstract

This paper presents an instruction-based coordination architecture for Field-Programmable Gate Array (FPGA)-based systems with multiple high-performance Processing Units (PUs) for accelerating Deep Neural Network (DNN) inference. This architecture enables programmable multi-PU synchronization through instruction controller units coupled with peer-to-peer instruction synchronization units, utilizing instruction types organized into load, compute, and store functional groups. A compilation framework is presented that transforms DNN models into executable instruction programs, enabling flexible partitioning of DNN models into topologically contiguous subgraphs mapped to available PUs. Multiple deployment strategies are supported, enabling pipeline parallelism among PUs and batch-level parallelism across different PU subsets, with runtime switching among them without FPGA reconfiguration. The proposed approach enables design space exploration, supporting dynamic trade-offs between single-batch and multi-batch performance. Experimental results on ResNet-50 demonstrate notable compute efficiency, up to $98\%$, and throughput efficiency gains, up to $2.7\times$, over prior works across different configurations.