Growing an architecture for a neural network

TL;DR

This paper introduces an automatic architecture search algorithm that iteratively prunes connections and adds neurons, capable of designing arbitrary graph-structured neural networks that outperform standard architectures.

Contribution

The paper presents a novel architecture search method that is not limited to layered structures, allowing for more flexible neural network designs.

Findings

Optimized networks outperform standard solutions in brightness prediction tasks.

The algorithm effectively minimizes complexity while maintaining accuracy.

Demonstrated on image brightness prediction and function approximation problems.

Abstract

We propose a new kind of automatic architecture search algorithm. The algorithm alternates pruning connections and adding neurons, and it is not restricted to layered architectures only. Here architecture is an arbitrary oriented graph with some weights (along with some biases and an activation function), so there may be no layered structure in such a network. The algorithm minimizes the complexity of staying within a given error. We demonstrate our algorithm on the brightness prediction problem of the next point through the previous points on an image. Our second test problem is the approximation of the bivariate function defining the brightness of a black and white image. Our optimized networks significantly outperform the standard solution for neural network architectures in both cases.