Neural Velocity for hyperparameter tuning

TL;DR

This paper introduces NeVe, a dynamic training method that uses neural velocity, the rate of change of neuron transfer functions, to improve hyperparameter tuning by reducing reliance on validation data.

Contribution

The paper proposes neural velocity as a novel metric for adaptive hyperparameter tuning, enabling more efficient training without extensive validation datasets.

Findings

Neural velocity effectively indicates model convergence.

Sampling neural velocity can be done with network noise, reducing validation dependence.

NeVe improves training efficiency through dynamic hyperparameter adjustment.

Abstract

Hyperparameter tuning, such as learning rate decay and defining a stopping criterion, often relies on monitoring the validation loss. This paper presents NeVe, a dynamic training approach that adjusts the learning rate and defines the stop criterion based on the novel notion of "neural velocity". The neural velocity measures the rate of change of each neuron's transfer function and is an indicator of model convergence: sampling neural velocity can be performed even by forwarding noise in the network, reducing the need for a held-out dataset. Our findings show the potential of neural velocity as a key metric for optimizing neural network training efficiently