Scalable Machines with Intrinsic Higher Mental-State Dynamics

TL;DR

This paper introduces a biologically inspired, scalable neural model that enhances computational efficiency and learning speed by mimicking higher mental-state dynamics, applicable to vision tasks like ImageNet classification.

Contribution

It presents a novel mathematically grounded framework linking neurobiological insights to scalable transformer models with improved efficiency.

Findings

Faster learning on ImageNet-1K compared to standard ViT

Reduced computational demand with fewer model components

Operates at approximately linear complexity with input size

Abstract

Drawing on recent breakthroughs in cellular neurobiology and detailed biophysical modeling linking neocortical pyramidal neurons to distinct mental-state regimes, this work introduces a mathematically grounded formulation showing how models (e.g., Transformers) can implement computational principles underlying awake imaginative thought to pre-select relevant information before attention is applied via triadic modulation loops among queries ($Q$), keys ($K$), and values ($V$).~Scalability experiments on ImageNet-1K, benchmarked against a standard Vision Transformer (ViT), demonstrate significantly faster learning with reduced computational demand (fewer heads, layers, and tokens), consistent with our prior findings in reinforcement learning and language modeling. The approach operates at approximately $\mathcal{O}(N)$ complexity with respect to the number of input tokens $N$.