A recurrent vision transformer shows signatures of primate visual attention

TL;DR

This paper introduces a Recurrent Vision Transformer that combines self-attention with recurrent memory, capturing key features of primate visual attention and demonstrating primate-like attention signatures in a computational model.

Contribution

The paper presents a novel Recurrent Vision Transformer integrating recurrent memory with self-attention, trained on a primate-inspired task, showing primate-like attention behaviors and neural signatures.

Findings

Model exhibits primate-like attention signatures

Attention maps show dynamic spatial prioritization

Perturbations mimic primate neural responses

Abstract

Attention is fundamental to both biological and artificial intelligence, yet research on animal attention and AI self attention remains largely disconnected. We propose a Recurrent Vision Transformer (Recurrent ViT) that integrates self-attention with recurrent memory, allowing both current inputs and stored information to guide attention allocation. Trained solely via sparse reward feedback on a spatially cued orientation change detection task, a paradigm used in primate studies, our model exhibits primate like signatures of attention, including improved accuracy and faster responses for cued stimuli that scale with cue validity. Analysis of self-attention maps reveals dynamic spatial prioritization with reactivation prior to expected changes, and targeted perturbations produce performance shifts similar to those observed in primate frontal eye fields and superior colliculus. These findings demonstrate that incorporating recurrent feedback into self attention can capture key aspects of primate visual attention.