Neural networks adapting to datasets: learning network size and topology

TL;DR

This paper presents a method for neural networks to learn their size and topology during training, resulting in task-specific graph structures that encode dataset characteristics.

Contribution

It introduces a flexible training setup enabling neural networks to adapt their structure dynamically, which is a novel approach to architecture learning during standard training.

Findings

Networks can be trained from scratch with similar performance.

The resulting architectures encode dataset-specific properties.

Systematic regularities observed across different datasets.

Abstract

We introduce a flexible setup allowing for a neural network to learn both its size and topology during the course of a standard gradient-based training. The resulting network has the structure of a graph tailored to the particular learning task and dataset. The obtained networks can also be trained from scratch and achieve virtually identical performance. We explore the properties of the network architectures for a number of datasets of varying difficulty observing systematic regularities. The obtained graphs can be therefore understood as encoding nontrivial characteristics of the particular classification tasks.