Intelligent Neural Networks: From Layered Architectures to Graph-Organized Intelligence

TL;DR

This paper introduces Intelligent Neural Networks (INN), a new graph-structured neural architecture where neurons have internal memory and learned communication, achieving superior performance and stability on character modeling tasks.

Contribution

The paper proposes a novel neuron-centric, graph-organized neural network architecture that enhances training stability and performance, differing from traditional layered models.

Findings

INN outperforms Transformers and LSTMs on Text8 benchmark.

Graph topology improves training stability over stacked Mamba blocks.

Ablation shows learned communication is crucial for performance.

Abstract

Biological neurons exhibit remarkable intelligence: they maintain internal states, communicate selectively with other neurons, and self-organize into complex graphs rather than rigid hierarchical layers. What if artificial intelligence could emerge from similarly intelligent computational units? We introduce Intelligent Neural Networks (INN), a paradigm shift where neurons are first-class entities with internal memory and learned communication patterns, organized in complete graphs rather than sequential layers. Each Intelligent Neuron combines selective state-space dynamics (knowing when to activate) with attention-based routing (knowing to whom to send signals), enabling emergent computation through graph-structured interactions. On the standard Text8 character modeling benchmark, INN achieves 1.705 Bit-Per-Character (BPC), significantly outperforming a comparable Transformer (2.055 BPC) and matching a highly optimized LSTM baseline. Crucially, a parameter-matched baseline of stacked Mamba blocks fails to converge (>3.4 BPC) under the same training protocol, demonstrating that INN's graph topology provides essential training stability. Ablation studies confirm this: removing inter-neuron communication degrades performance or leads to instability, proving the value of learned neural routing. This work demonstrates that neuron-centric design with graph organization is not merely bio-inspired -- it is computationally effective, opening new directions for modular, interpretable, and scalable neural architectures.