TransAxx: Efficient Transformers with Approximate Computing

TL;DR

TransAxx introduces a framework for evaluating and optimizing Vision Transformer models with approximate computing techniques, enabling significant power savings while maintaining accuracy on low-power devices.

Contribution

The paper presents TransAxx, a novel PyTorch-based framework that integrates approximate arithmetic into ViT models and proposes a Monte Carlo Tree Search method for generating efficient approximate accelerators.

Findings

Approximate multipliers can be effectively used in ViT models.

Approximate-aware finetuning recovers accuracy lost due to approximation.

Significant power savings achieved with minimal accuracy loss.

Abstract

Vision Transformer (ViT) models which were recently introduced by the transformer architecture have shown to be very competitive and often become a popular alternative to Convolutional Neural Networks (CNNs). However, the high computational requirements of these models limit their practical applicability especially on low-power devices. Current state-of-the-art employs approximate multipliers to address the highly increased compute demands of DNN accelerators but no prior research has explored their use on ViT models. In this work we propose TransAxx, a framework based on the popular PyTorch library that enables fast inherent support for approximate arithmetic to seamlessly evaluate the impact of approximate computing on DNNs such as ViT models. Using TransAxx we analyze the sensitivity of transformer models on the ImageNet dataset to approximate multiplications and perform approximate-aware finetuning to regain accuracy. Furthermore, we propose a methodology to generate approximate accelerators for ViT models. Our approach uses a Monte Carlo Tree Search (MCTS) algorithm to efficiently search the space of possible configurations using a hardware-driven hand-crafted policy. Our evaluation demonstrates the efficacy of our methodology in achieving significant trade-offs between accuracy and power, resulting in substantial gains without compromising on performance.