Emergent Structures and Lifetime Structure Evolution in Artificial Neural Networks

TL;DR

This paper introduces the Unstructured Recursive Network (URN), a flexible neural network model that dynamically evolves its structure during training, mimicking biological neural adaptability and unifying various architectures through a common loss function.

Contribution

The paper presents URN, a novel neural network framework that can generate diverse structures during training, bridging different architectures with a single loss function and natural symmetry considerations.

Findings

Different neural network architectures can emerge from URN during training.

The structure of the emergent network depends on data and regulator terms.

The loss function and regulators are derived from network symmetries and data geometry.

Abstract

Motivated by the flexibility of biological neural networks whose connectivity structure changes significantly during their lifetime, we introduce the Unstructured Recursive Network (URN) and demonstrate that it can exhibit similar flexibility during training via gradient descent. We show empirically that many of the different neural network structures commonly used in practice today (including fully connected, locally connected and residual networks of different depths and widths) can emerge dynamically from the same URN. These different structures can be derived using gradient descent on a single general loss function where the structure of the data and the relative strengths of various regulator terms determine the structure of the emergent network. We show that this loss function and the regulators arise naturally when considering the symmetries of the network as well as the geometric properties of the input data.