Neurons learn slower than they think

TL;DR

This paper investigates the complex dynamics of convergence in gradient-based learning, revealing that faster convergence can hinder test accuracy growth and that controlling convergence rates affects the model's differential capability.

Contribution

It introduces the concept of differential capability to analyze how convergence rates influence test accuracy growth in classification models.

Findings

Higher convergence rates slow down capability growth.

Lower convergence rates accelerate capability growth and decay.

Regulating convergence rates impacts differential capability.

Abstract

Recent studies revealed complex convergence dynamics in gradient-based methods, which has been little understood so far. Changing the step size to balance between high convergence rate and small generalization error may not be sufficient: maximizing the test accuracy usually requires a larger learning rate than minimizing the training loss. To explore the dynamic bounds of convergence rate, this study introduces \textit{differential capability} into an optimization process, which measures whether the test accuracy increases as fast as a model approaches the decision boundary in a classification problem. The convergence analysis showed that: 1) a higher convergence rate leads to slower capability growth; 2) a lower convergence rate results in faster capability growth and decay; 3) regulating a convergence rate in either direction reduces differential capability.