A Memory-Efficient Framework for Deformable Transformer with Neural Architecture Search

TL;DR

This paper introduces a hardware-efficient framework for deformable attention transformers using neural architecture search, significantly reducing memory access and maintaining high accuracy for edge hardware deployment.

Contribution

It proposes a NAS-based slicing strategy for DAT that optimizes hardware cost and accuracy without altering the model architecture.

Findings

Maintains only 0.2% accuracy drop compared to baseline DAT.

Reduces DRAM access times to 18% on FPGA hardware.

Demonstrates effective hardware acceleration on edge devices.

Abstract

Deformable Attention Transformers (DAT) have shown remarkable performance in computer vision tasks by adaptively focusing on informative image regions. However, their data-dependent sampling mechanism introduces irregular memory access patterns, posing significant challenges for efficient hardware deployment. Existing acceleration methods either incur high hardware overhead or compromise model accuracy. To address these issues, this paper proposes a hardware-friendly optimization framework for DAT. First, a neural architecture search (NAS)-based method with a new slicing strategy is proposed to automatically divide the input feature into uniform patches during the inference process, avoiding memory conflicts without modifying model architecture. The method explores the optimal slice configuration by jointly optimizing hardware cost and inference accuracy. Secondly, an FPGA-based verification system is designed to test the performance of this framework on edge-side hardware. Algorithm experiments on the ImageNet-1K dataset demonstrate that our hardware-friendly framework can maintain have only 0.2% accuracy drop compared to the baseline DAT. Hardware experiments on Xilinx FPGA show the proposed method reduces DRAM access times to 18% compared with existing DAT acceleration methods.