Element-Wise Attention Layers: an option for optimization

TL;DR

This paper introduces element-wise attention layers that adapt dot-product attention to reduce parameter count, maintaining competitive accuracy on image classification tasks with significantly fewer parameters, thus optimizing model efficiency.

Contribution

It proposes a novel element-wise attention mechanism that simplifies dot-product attention, enabling more efficient models with fewer parameters without substantial accuracy loss.

Findings

Achieved 92% accuracy on Fashion MNIST with 97% fewer parameters.

Maintained 60% accuracy on CIFAR10 with 50% fewer parameters.

Demonstrated effectiveness on VGG-like architectures using standard datasets.

Abstract

The use of Attention Layers has become a trend since the popularization of the Transformer-based models, being the key element for many state-of-the-art models that have been developed through recent years. However, one of the biggest obstacles in implementing these architectures - as well as many others in Deep Learning Field - is the enormous amount of optimizing parameters they possess, which make its use conditioned on the availability of robust hardware. In this paper, it's proposed a new method of attention mechanism that adapts the Dot-Product Attention, which uses matrices multiplications, to become element-wise through the use of arrays multiplications. To test the effectiveness of such approach, two models (one with a VGG-like architecture and one with the proposed method) have been trained in a classification task using Fashion MNIST and CIFAR10 datasets. Each model has been trained for 10 epochs in a single Tesla T4 GPU from Google Colaboratory. The results show that this mechanism allows for an accuracy of 92% of the VGG-like counterpart in Fashion MNIST dataset, while reducing the number of parameters in 97%. For CIFAR10, the accuracy is still equivalent to 60% of the VGG-like counterpart while using 50% less parameters.