Understanding Attention: In Minds and Machines

TL;DR

This paper reviews the concept of attention across disciplines, highlighting its role in resource management in neural networks and neuroscience, and advocates for unified frameworks to advance understanding in AI and brain sciences.

Contribution

It provides a comprehensive review of attention in artificial neural networks and neuroscience, proposing a unified conceptual framework for interdisciplinary understanding.

Findings

Attention involves adaptive resource control across fields

Neuroscience and AI share conceptual parallels in attention

Unified frameworks can facilitate cross-disciplinary insights

Abstract

Attention is a complex and broad concept, studied across multiple disciplines spanning artificial intelligence, cognitive science, psychology, neuroscience, and related fields. Although many of the ideas regarding attention do not significantly overlap among these fields, there is a common theme of adaptive control of limited resources. In this work, we review the concept and variants of attention in artificial neural networks (ANNs). We also discuss the origin of attention from the neuroscience point of view parallel to that of ANNs. Instead of having seemingly disconnected dialogues between varied disciplines, we suggest grounding the ideas on common conceptual frameworks for a systematic analysis of attention and towards possible unification of ideas in AI and Neuroscience.