The BUTTER Zone: An Empirical Study of Training Dynamics in Fully Connected Neural Networks

TL;DR

This paper introduces a comprehensive empirical dataset of training dynamics for fully-connected neural networks, enabling large-scale analysis of hyperparameters and performance patterns to foster scientific understanding in deep learning.

Contribution

It provides a large, publicly available dataset of 11 million training runs across diverse hyperparameters, facilitating systematic analysis of neural network training behaviors.

Findings

Identification of durable training patterns across tasks and architectures

Insights into hyperparameter effects on training and generalization

Large-scale dataset enabling future theoretical research

Abstract

We present an empirical dataset surveying the deep learning phenomenon on fully-connected feed-forward multilayer perceptron neural networks. The dataset, which is now freely available online, records the per-epoch training and generalization performance of 483 thousand distinct hyperparameter choices of architectures, tasks, depths, network sizes (number of parameters), learning rates, batch sizes, and regularization penalties. Repeating each experiment an average of 24 times resulted in 11 million total training runs and 40 billion epochs recorded. Accumulating this 1.7 TB dataset utilized 11 thousand CPU core-years, 72.3 GPU-years, and 163 node-years. In surveying the dataset, we observe durable patterns persisting across tasks and topologies. We aim to spark scientific study of machine learning techniques as a catalyst for the theoretical discoveries needed to progress the field beyond energy-intensive and heuristic practices.