Accelerating Transposed Convolutions on FPGA-based Edge Devices

TL;DR

This paper introduces MM2IM, a hardware-software co-designed accelerator for transposed convolutions on FPGA-based edge devices, significantly improving speed and energy efficiency over existing methods.

Contribution

The paper presents MM2IM, a novel accelerator combining matrix multiplication with col2IM, optimized for resource-constrained edge devices to enhance transposed convolution performance.

Findings

Achieves 1.9x average speedup over CPU baseline.

Up to 4.2x speedup on generative model layers.

Up to 3x speedup and 2.4x energy reduction on GAN models.

Abstract

Transposed Convolutions (TCONV) enable the up-scaling mechanism within generative Artificial Intelligence (AI) models. However, the predominant Input-Oriented Mapping (IOM) method for implementing TCONV has complex output mapping, overlapping sums, and ineffectual computations. These inefficiencies further exacerbate the performance bottleneck of TCONV and generative models on resource-constrained edge devices. To address this problem, in this paper we propose MM2IM, a hardware-software co-designed accelerator that combines Matrix Multiplication (MatMul) with col2IM to process TCONV layers on resource-constrained edge devices efficiently. Using the SECDA-TFLite design toolkit, we implement MM2IM and evaluate its performance across 261 TCONV problem configurations, achieving an average speedup of 1.9x against a dual-thread ARM Neon optimized CPU baseline. We then evaluate the performance of MM2IM on a range of TCONV layers from well-known generative models achieving up to 4.2x speedup, and compare it against similar resource-constrained TCONV accelerators, outperforming them by at least 2x GOPs/DSP. Finally, we evaluate MM2IM on the DCGAN and pix2pix GAN models, achieving up to 3x speedup and 2.4x energy reduction against the CPU baseline.