Beyond Attention: Toward Machines with Intrinsic Higher Mental States

TL;DR

This paper proposes a neuro-inspired approach for machine learning models like Transformers to emulate higher mental states, enabling faster learning and more efficient reasoning by pre-selecting relevant information through neuronal-level modulation.

Contribution

It introduces a novel triadic modulation mechanism inspired by neurobiology that enhances reasoning and reduces computational costs in Transformer models.

Findings

Achieved faster learning with fewer model components.

Demonstrated effectiveness in reinforcement learning, vision, and NLP tasks.

Reduced computational demand by orders of magnitude.

Abstract

Attending to what is relevant is fundamental to both the mammalian brain and modern machine learning models such as Transformers. Yet, determining relevance remains a core challenge, traditionally offloaded to learning algorithms like backpropagation. Inspired by recent cellular neurobiological evidence linking neocortical pyramidal cells to distinct mental states, this work shows how models (e.g., Transformers) can emulate high-level perceptual processing and awake thought (imagination) states to pre-select relevant information before applying attention. Triadic neuronal-level modulation loops among questions ($Q$), clues (keys, $K$), and hypotheses (values, $V$) enable diverse, deep, parallel reasoning chains at the representation level and allow a rapid shift from initial biases to refined understanding. This leads to orders-of-magnitude faster learning with significantly reduced computational demand (e.g., fewer heads, layers, and tokens), at an approximate cost of $\mathcal{O}(N)$, where $N$ is the number of input tokens. Results span reinforcement learning (e.g., CarRacing in a high-dimensional visual setup), computer vision, and natural language question answering.