Attention Mechanism in Neural Networks: Where it Comes and Where it Goes

TL;DR

This paper reviews the evolution of attention mechanisms in neural networks, highlighting key milestones and recent advancements, to guide future research and inspire novel approaches beyond current attention models.

Contribution

It provides a comprehensive overview of the development of attention mechanisms from inception to recent trends, serving as a roadmap for future exploration.

Findings

Attention mechanisms have evolved significantly over time.

Recent models demonstrate remarkable performance improvements.

The review highlights key milestones across various tasks.

Abstract

A long time ago in the machine learning literature, the idea of incorporating a mechanism inspired by the human visual system into neural networks was introduced. This idea is named the attention mechanism, and it has gone through a long development period. Today, many works have been devoted to this idea in a variety of tasks. Remarkable performance has recently been demonstrated. The goal of this paper is to provide an overview from the early work on searching for ways to implement attention idea with neural networks until the recent trends. This review emphasizes the important milestones during this progress regarding different tasks. By this way, this study aims to provide a road map for researchers to explore the current development and get inspired for novel approaches beyond the attention.