ViT-LCA: A Neuromorphic Approach for Vision Transformers

TL;DR

This paper introduces ViT-LCA, a neuromorphic-compatible vision transformer model that improves accuracy and energy efficiency on ImageNet-1K, enabling better deployment on neuromorphic hardware.

Contribution

It presents a novel combination of vision transformers with the Locally Competitive Algorithm for efficient neuromorphic implementation.

Findings

Higher accuracy on ImageNet-1K

Significantly lower energy consumption

Better compatibility with neuromorphic hardware

Abstract

The recent success of Vision Transformers has generated significant interest in attention mechanisms and transformer architectures. Although existing methods have proposed spiking self-attention mechanisms compatible with spiking neural networks, they often face challenges in effective deployment on current neuromorphic platforms. This paper introduces a novel model that combines vision transformers with the Locally Competitive Algorithm (LCA) to facilitate efficient neuromorphic deployment. Our experiments show that ViT-LCA achieves higher accuracy on ImageNet-1K dataset while consuming significantly less energy than other spiking vision transformer counterparts. Furthermore, ViT-LCA's neuromorphic-friendly design allows for more direct mapping onto current neuromorphic architectures.