Neural Capacitance: A New Perspective of Neural Network Selection via Edge Dynamics

TL;DR

This paper introduces a neural capacitance metric derived from edge dynamics during training, enabling efficient early prediction of a neural network's generalization ability for downstream tasks, reducing the need for extensive training.

Contribution

The paper proposes a novel framework that models neural network training as edge dynamics and introduces a neural capacitance metric for early performance prediction.

Findings

Neural capacitance correlates strongly with final model performance.

The method outperforms existing early prediction techniques.

Applicable to various models and datasets with high efficiency.

Abstract

Efficient model selection for identifying a suitable pre-trained neural network to a downstream task is a fundamental yet challenging task in deep learning. Current practice requires expensive computational costs in model training for performance prediction. In this paper, we propose a novel framework for neural network selection by analyzing the governing dynamics over synaptic connections (edges) during training. Our framework is built on the fact that back-propagation during neural network training is equivalent to the dynamical evolution of synaptic connections. Therefore, a converged neural network is associated with an equilibrium state of a networked system composed of those edges. To this end, we construct a network mapping $\phi$, converting a neural network $G_A$ to a directed line graph $G_B$ that is defined on those edges in $G_A$. Next, we derive a neural capacitance metric $\beta_{\rm eff}$ as a predictive measure universally capturing the generalization capability of $G_A$ on the downstream task using only a handful of early training results. We carried out extensive experiments using 17 popular pre-trained ImageNet models and five benchmark datasets, including CIFAR10, CIFAR100, SVHN, Fashion MNIST and Birds, to evaluate the fine-tuning performance of our framework. Our neural capacitance metric is shown to be a powerful indicator for model selection based only on early training results and is more efficient than state-of-the-art methods.