A framework for measuring the training efficiency of a neural architecture

TL;DR

This paper introduces an experimental framework to measure neural architecture training efficiency, revealing how efficiency varies with training progress, stopping criteria, and model complexity across CNNs and Bayesian models.

Contribution

The paper proposes a novel framework for assessing training efficiency and demonstrates its application on CNNs and Bayesian models with insights into efficiency decay and architecture comparison.

Findings

Training efficiency decays as training progresses.

CNNs are more efficient than Bayesian CNNs on MNIST and CIFAR-10.

Efficiency differences become more pronounced with increased task complexity.

Abstract

Measuring Efficiency in neural network system development is an open research problem. This paper presents an experimental framework to measure the training efficiency of a neural architecture. To demonstrate our approach, we analyze the training efficiency of Convolutional Neural Networks and Bayesian equivalents on the MNIST and CIFAR-10 tasks. Our results show that training efficiency decays as training progresses and varies across different stopping criteria for a given neural model and learning task. We also find a non-linear relationship between training stopping criteria, training Efficiency, model size, and training Efficiency. Furthermore, we illustrate the potential confounding effects of overtraining on measuring the training efficiency of a neural architecture. Regarding relative training efficiency across different architectures, our results indicate that CNNs are more efficient than BCNNs on both datasets. More generally, as a learning task becomes more complex, the relative difference in training efficiency between different architectures becomes more pronounced.