Activator: GLU Activation Function as the Core Component of a Vision Transformer

TL;DR

This paper proposes replacing the traditional attention mechanism in vision transformers with a GLU-based architecture to reduce computational costs while maintaining competitive performance.

Contribution

It introduces a novel transformer architecture using GLU activation functions, offering a more efficient alternative to standard attention-based models in vision tasks.

Findings

GLU-based architecture reduces computational complexity.

Competitive performance achieved compared to baseline models.

Supports more efficient vision transformer designs.

Abstract

The transformer architecture has driven many successes in a variety of tasks within the field of deep learning, in particular the recent advances in natural language processing (NLP) culminating with large language models (LLM). Adding to that success, transformer architecture has found widespread interest from computer vision (CV) researchers and practitioners, allowing for many advancements in vision-related tasks and opening the door for multitask and multi-modal deep learning architectures that share the same principle of operation. One drawback to these architectures is their reliance on the scaled dot product attention mechanism with the softmax activation function, which is computationally expensive and requires large compute capabilities for both training and inference. This paper investigates substituting the MLP and attention mechanism usually adopted for transformer architecture with an architecture based on incorporating a gated linear unit (GLU) activation function structure with the aim of reducing the computational cost. The equalized experimental assessments conducted in this work show that the proposed modification with the targeted reductions in computational complexity offers competitive performance compared to the selected baseline architectures. The results are significantly in support of the aims of this work, in which the focus was to extensively utilize GLU-based MLPs, establishing a more efficient but capable alternative to the traditional MLP and the attention mechanism as the core component in the design of transformer architectures.