Integration of a systolic array based hardware accelerator into a DNN operator auto-tuning framework

TL;DR

This paper presents the integration of a systolic array-based hardware accelerator with a DNN operator auto-tuning framework, demonstrating improved performance and flexibility for deploying neural networks on heterogeneous systems.

Contribution

It introduces a generic schedule for GEMM operations on Gemmini and integrates it with TVM, enabling automatic performance tuning and outperforming previous hand-tuned schedules.

Findings

Achieved 46 GOPs throughput on FPGA at 100 MHz

Generated code outperforms default hand-tuned schedules

Successful integration of Gemmini with TVM for auto-tuning

Abstract

The deployment of neural networks on heterogeneous SoCs coupled with custom accelerators is a challenging task because of the lack of end-to-end software tools provided for these systems. Moreover, the already available low level schedules and mapping strategies provided by the accelerator developers for typical tensor operations are not necessarily the best possible ones for each particular use case. This is why frameworks which automatically test the performance of the generated code on a specific hardware configuration are of special interest. In this work, the integration between the code generation framework TVM and the systolic array-based accelerator Gemmini is presented. A generic schedule to offload the GEneral Matrix Multiply (GEMM) tensor operation onto Gemmini is detailed, and its suitability is tested by executing the AutoTVM tuning process on it. Our generated code achieves a peak throughput of 46 giga-operations per second (GOPs) under a 100 MHz clock on a Xilinx ZCU102 FPGA, outperforming previous work. Furthermore, the code generated by this integration was able to surpass the default hand-tuned schedules provided by the Gemmini developers in real-world workloads.