Architecture Agnostic Neural Networks

TL;DR

This paper investigates the possibility of creating neural networks that are architecture agnostic by mimicking the brain's stochastic synaptic pruning, using sparse binary networks to find high-performing, task-capable network families without traditional training.

Contribution

The study introduces a method to synthesize architecture agnostic neural networks using a manifold search in sparse binary networks, bypassing backpropagation for training.

Findings

High-performing architecture families share similar sparsity and weight distributions.

Architecture agnostic networks succeed in static and dynamic tasks.

Proposed method discovers diverse neural network architectures without training.

Abstract

In this paper, we explore an alternate method for synthesizing neural network architectures, inspired by the brain's stochastic synaptic pruning. During a person's lifetime, numerous distinct neuronal architectures are responsible for performing the same tasks. This indicates that biological neural networks are, to some degree, architecture agnostic. However, artificial networks rely on their fine-tuned weights and hand-crafted architectures for their remarkable performance. This contrast begs the question: Can we build artificial architecture agnostic neural networks? To ground this study we utilize sparse, binary neural networks that parallel the brain's circuits. Within this sparse, binary paradigm we sample many binary architectures to create families of architecture agnostic neural networks not trained via backpropagation. These high-performing network families share the same sparsity, distribution of binary weights, and succeed in both static and dynamic tasks. In summation, we create an architecture manifold search procedure to discover families or architecture agnostic neural networks.