From Cognition to Computation: A Comparative Review of Human Attention and Transformer Architectures

TL;DR

This paper compares human attention and Transformer architectures, highlighting their differences and similarities to inform future AI development through interdisciplinary insights.

Contribution

It provides a comprehensive comparative review of human and artificial attention mechanisms from a cognitive-functional perspective.

Findings

Human and Transformer attention differ in capacity constraints

Attention pathways and intentional mechanisms vary between humans and models

The review identifies open research questions for AI advancement

Abstract

Attention is a cornerstone of human cognition that facilitates the efficient extraction of information in everyday life. Recent developments in artificial intelligence like the Transformer architecture also incorporate the idea of attention in model designs. However, despite the shared fundamental principle of selectively attending to information, human attention and the Transformer model display notable differences, particularly in their capacity constraints, attention pathways, and intentional mechanisms. Our review aims to provide a comparative analysis of these mechanisms from a cognitive-functional perspective, thereby shedding light on several open research questions. The exploration encourages interdisciplinary efforts to derive insights from human attention mechanisms in the pursuit of developing more generalized artificial intelligence.