A Local Optima Network Analysis of the Feedforward Neural Architecture Space

TL;DR

This paper applies local optima network analysis to the space of small feedforward neural architectures, revealing simple global structures that could inform architecture optimization strategies.

Contribution

It introduces the use of LON analysis for neural architecture space characterization, providing insights into the global landscape structure of small neural networks.

Findings

LONs exhibit simple global funnel structures in most cases

The neural architecture space has heterogeneous but predominantly funnel-shaped landscapes

LON analysis could be a promising tool for neural architecture search

Abstract

This study investigates the use of local optima network (LON) analysis, a derivative of the fitness landscape of candidate solutions, to characterise and visualise the neural architecture space. The search space of feedforward neural network architectures with up to three layers, each with up to 10 neurons, is fully enumerated by evaluating trained model performance on a selection of data sets. Extracted LONs, while heterogeneous across data sets, all exhibit simple global structures, with single global funnels in all cases but one. These results yield early indication that LONs may provide a viable paradigm by which to analyse and optimise neural architectures.