Resonant Sparse Geometry Networks

TL;DR

Resonant Sparse Geometry Networks (RSGN) are a brain-inspired architecture that employs input-dependent sparse connectivity in hyperbolic space, achieving high accuracy with significantly fewer parameters and lower computational complexity than traditional Transformers.

Contribution

The paper introduces RSGN, a novel sparse, hierarchical neural network architecture with input-dependent connectivity and dual timescale learning, inspired by brain principles.

Findings

Achieves 96.5% accuracy on long-range dependency tasks with 15x fewer parameters than Transformers.

Attains 23.8% accuracy on hierarchical classification with 41,672 parameters, outperforming larger Transformer baselines.

Demonstrates O(n*k) computational complexity, significantly more efficient than dense attention mechanisms.

Abstract

We introduce Resonant Sparse Geometry Networks (RSGN), a brain-inspired architecture with self-organizing sparse hierarchical input-dependent connectivity. Unlike Transformer architectures that employ dense attention mechanisms with O(n^2) computational complexity, RSGN embeds computational nodes in learned hyperbolic space where connection strength decays with geodesic distance, achieving dynamic sparsity that adapts to each input. The architecture operates on two distinct timescales: fast differentiable activation propagation optimized through gradient descent, and slow Hebbian-inspired structural learning for connectivity adaptation through local correlation rules. We provide rigorous mathematical analysis demonstrating that RSGN achieves O(n*k) computational complexity, where k << n represents the average active neighborhood size. Experimental evaluation on hierarchical classification and long-range dependency tasks demonstrates that RSGN achieves 96.5% accuracy on long-range dependency tasks while using approximately 15x fewer parameters than standard Transformers. On challenging hierarchical classification with 20 classes, RSGN achieves 23.8% accuracy (compared to 5% random baseline) with only 41,672 parameters, nearly 10x fewer than the Transformer baselines which require 403,348 parameters to achieve 30.1% accuracy. Our ablation studies confirm the contribution of each architectural component, with Hebbian learning providing consistent improvements. These results suggest that brain-inspired principles of sparse, geometrically-organized computation offer a promising direction toward more efficient and biologically plausible neural architectures.