A scalable constructive algorithm for the optimization of neural network architectures

TL;DR

This paper introduces a scalable greedy algorithm for neural network architecture optimization that efficiently finds minimal-layer networks with competitive accuracy, outperforming existing hyperparameter search methods in performance and speed.

Contribution

The paper presents a novel scalable greedy search algorithm for neural network architecture optimization that improves efficiency and predictive performance over existing hyperparameter tuning methods.

Findings

Outperforms state-of-the-art hyperparameter optimization algorithms in accuracy.

Reduces time-to-solution for architecture search.

Finds minimal-layer networks with comparable or better performance.

Abstract

We propose a new scalable method to optimize the architecture of an artificial neural network. The proposed algorithm, called Greedy Search for Neural Network Architecture, aims to determine a neural network with minimal number of layers that is at least as performant as neural networks of the same structure identified by other hyperparameter search algorithms in terms of accuracy and computational cost. Numerical results performed on benchmark datasets show that, for these datasets, our method outperforms state-of-the-art hyperparameter optimization algorithms in terms of attainable predictive performance by the selected neural network architecture, and time-to-solution for the hyperparameter optimization to complete.