Training Neural Networks by Optimizing Neuron Positions

TL;DR

This paper introduces a neural network architecture where neurons are embedded in space and their connections are based on inverse distance, reducing parameters and maintaining competitive performance, inspired by biological wiring principles.

Contribution

The paper presents a novel spatially embedded neural network model that replaces traditional weights with distance-based connections, improving parameter efficiency and biological plausibility.

Findings

Achieves competitive performance on MNIST with fewer parameters.

Maintains accuracy at over 80% sparsity, outperforming traditional networks.

Provides intuitive visualization of network structure through spatial embedding.

Abstract

The high computational complexity and increasing parameter counts of deep neural networks pose significant challenges for deployment in resource-constrained environments, such as edge devices or real-time systems. To address this, we propose a parameter-efficient neural architecture where neurons are embedded in Euclidean space. During training, their positions are optimized and synaptic weights are determined as the inverse of the spatial distance between connected neurons. These distance-dependent wiring rules replace traditional learnable weight matrices and significantly reduce the number of parameters while introducing a biologically inspired inductive bias: connection strength decreases with spatial distance, reflecting the brain's embedding in three-dimensional space where connections tend to minimize wiring length. We validate this approach for both multi-layer perceptrons and spiking neural networks. Through a series of experiments, we demonstrate that these spatially embedded neural networks achieve a performance competitive with conventional architectures on the MNIST dataset. Additionally, the models maintain performance even at pruning rates exceeding 80% sparsity, outperforming traditional networks with the same number of parameters under similar conditions. Finally, the spatial embedding framework offers an intuitive visualization of the network structure.