Evolving Layer-Specific Scalar Functions for Hardware-Aware Transformer Adaptation

TL;DR

This paper introduces a hardware-aware, evolutionary approach to adapt Vision Transformers for edge devices by evolving layer-specific scalar functions, significantly reducing complexity while maintaining high accuracy.

Contribution

It presents a novel genetic programming framework to generate layer-specific scalar functions, eliminating the need for retraining and improving hardware efficiency.

Findings

Achieves 91.6% variance approximation of normalization behaviors.

Recovers 84.25% Top-1 accuracy on ImageNet-1K in 20 epochs.

Reduces global reduction bottleneck, enabling efficient edge deployment.

Abstract

Vision Transformers (ViTs) achieve state-of-the-art performance on challenging vision tasks, but their deployment on edge devices is severely hindered by the computational complexity and global reduction bottleneck imposed by layer normalization. Recent methods attempt to bypass this by replacing normalization layers with hardware-friendly scalar approximations. However, these homogeneous replacements do not optimally fit to all layers' behaviour and rely on expensive model retraining. In this work, we propose a highly efficient, hardware-aware framework that utilizes genetic programming (GP) to evolve heterogeneous, layer-specific scalar functions directly from pre-trained weights. Coupled with a novel post-training re-alignment strategy, our approach eliminates the need to retrain models from scratch entirely. Our evolved expressions accurately approximate the target normalization behaviours, capturing $91.6\%$ of the variance ($R^2$) compared to only $70.2\%$ for homogeneous baselines, allowing our modified architecture to recover $84.25\%$ Top-1 ImageNet-1K accuracy in only 20 epochs. By preserving this performance while eliminating the global reduction bottleneck, our approach establishes a highly favourable trade-off between arithmetic complexity and off-chip memory traffic, removing a primary barrier to the efficient deployment of ViTs on edge accelerators.