Measuring training variability from stochastic optimization using robust nonparametric testing

TL;DR

This paper introduces a robust hypothesis testing framework and a novel summary statistic, the alpha-trimming level, to measure training variability in stochastic neural network training more effectively than traditional metrics.

Contribution

It proposes a new statistical approach and summary statistic for assessing model similarity and variability, addressing limitations of existing metrics.

Findings

Alpha-trimming level effectively measures model variability.

Framework determines how many training runs are needed for reliable ensembles.

Method outperforms traditional metrics like accuracy and calibration error.

Abstract

Deep neural network training often involves stochastic optimization, meaning each run will produce a different model. This implies that hyperparameters of the training process, such as the random seed itself, can potentially have significant influence on the variability in the trained models. Measuring model quality by summary statistics, such as test accuracy, can obscure this dependence. We propose a robust hypothesis testing framework and a novel summary statistic, the $\alpha$-trimming level, to measure model similarity. Applying hypothesis testing directly with the $\alpha$-trimming level is challenging because we cannot accurately describe the distribution under the null hypothesis. Our framework addresses this issue by determining how closely an approximate distribution resembles the expected distribution of a group of individually trained models and using this approximation as our reference. We then use the $\alpha$-trimming level to suggest how many training runs should be sampled to ensure that an ensemble is a reliable representative of the true model performance. We also show how to use the $\alpha$-trimming level to measure model variability and demonstrate experimentally that it is more expressive than performance metrics like validation accuracy, churn, or expected calibration error when taken alone. An application of fine-tuning over random seed in transfer learning illustrates the advantage of our new metric.