Automated Architecture Synthesis for Arbitrarily Structured Neural Networks

TL;DR

This paper introduces a novel framework for neural network architecture synthesis that learns arbitrary graph structures inspired by biological neural systems, enabling enhanced communication, collaboration, and efficiency.

Contribution

It proposes a new method to learn and organize neural modules into arbitrary graph structures during training, surpassing traditional DAG-based NAS approaches.

Findings

Enables neural networks to learn complex, arbitrary graph structures.

Improves network efficiency and reduces overfitting through modular organization.

Demonstrates adaptability across various tasks and scenarios.

Abstract

This paper offers a new perspective on Artificial Neural Networks (ANNs) architecture. Traditional ANNs commonly use tree-like or DAG structures for simplicity, which can be preset or determined by Neural Architecture Search (NAS). Yet, these structures restrict network collaboration and capability due to the absence of horizontal and backward communication. Biological neural systems, however, feature billions of neural units with highly complex connections, allowing each biological neuron to connect with others based on specific situations. Inspired by biological systems, we propose a novel framework that learns to construct arbitrary graph structures during training and introduce the concept of Neural Modules for organizing neural units, which facilitates communication between any nodes and collaboration among modules. Unlike traditional NAS methods that rely on DAG search spaces, our framework learns from complete graphs, enabling free communication between neurons akin to biological neural networks. Furthermore, we present a method to compute these structures and a regularization technique that organizes them into multiple independent, balanced neural modules. This approach reduces overfitting and improves efficiency through parallel computing. Overall, our method allows ANNs to learn effective arbitrary structures similar to biological ones. It is adaptable to various tasks and compatible across different scenarios, with experimental results demonstrating its potential.