Attention, please! A survey of Neural Attention Models in Deep Learning

TL;DR

This survey comprehensively reviews neural attention models in deep learning, analyzing their architectures, applications, interpretability, and future research directions across various neural network types and domains.

Contribution

It systematically analyzes 650 works on neural attention, develops an automated review methodology, and provides insights into attention's impact and future trends in deep learning.

Findings

Attention significantly impacts convolutional, recurrent, and generative models.

Attention enhances interpretability of neural networks.

Identifies key trends and opportunities for future research.

Abstract

In humans, Attention is a core property of all perceptual and cognitive operations. Given our limited ability to process competing sources, attention mechanisms select, modulate, and focus on the information most relevant to behavior. For decades, concepts and functions of attention have been studied in philosophy, psychology, neuroscience, and computing. For the last six years, this property has been widely explored in deep neural networks. Currently, the state-of-the-art in Deep Learning is represented by neural attention models in several application domains. This survey provides a comprehensive overview and analysis of developments in neural attention models. We systematically reviewed hundreds of architectures in the area, identifying and discussing those in which attention has shown a significant impact. We also developed and made public an automated methodology to facilitate the development of reviews in the area. By critically analyzing 650 works, we describe the primary uses of attention in convolutional, recurrent networks and generative models, identifying common subgroups of uses and applications. Furthermore, we describe the impact of attention in different application domains and their impact on neural networks' interpretability. Finally, we list possible trends and opportunities for further research, hoping that this review will provide a succinct overview of the main attentional models in the area and guide researchers in developing future approaches that will drive further improvements.