GLU Attention Improve Transformer

TL;DR

This paper introduces GLU Attention, a lightweight, parameter-free attention mechanism that enhances transformer models' performance and convergence speed across text and vision tasks without adding computational costs.

Contribution

It proposes a novel GLU Attention mechanism that introduces nonlinearity into attention values, improving performance and compatibility with existing transformer variants.

Findings

Improves model performance in text and vision tasks

Speeds up convergence without extra parameters

Integrates seamlessly with existing attention methods

Abstract

Gated Linear Units (GLU) have shown great potential in enhancing neural network performance. In this paper, I introduce a novel attention mechanism called GLU Attention, which introduces nonlinearity into the values of Attention. My experiments demonstrate that GLU Attention improves both model performance and convergence speed across text and vision modalities with zero additional parameters and negligible computational costs. GLU Attention is lightweight and can seamlessly integrate with other technologies, such as Flash Attention, Rotary Position Embedding (RoPE), and various Multi-Head Attention (MHA) variants such as Grouped-Query Attention (GQA). This project is open-sourced at github.