Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Training

TL;DR

This paper introduces TempBalance, a layer-wise learning rate method inspired by temperature concepts and HT-SR theory, which improves neural network training performance across multiple datasets and architectures.

Contribution

It proposes TempBalance, a novel layer-wise learning rate scheduling method based on HT-SR theory, enhancing training effectiveness and outperforming existing optimizers and schedulers.

Findings

TempBalance improves test performance on CIFAR10, CIFAR100, SVHN, TinyImageNet.

It outperforms SGD, spectral norm regularization, and state-of-the-art optimizers.

Layer-wise temperature balancing leads to better model generalization.

Abstract

Regularization in modern machine learning is crucial, and it can take various forms in algorithmic design: training set, model family, error function, regularization terms, and optimizations. In particular, the learning rate, which can be interpreted as a temperature-like parameter within the statistical mechanics of learning, plays a crucial role in neural network training. Indeed, many widely adopted training strategies basically just define the decay of the learning rate over time. This process can be interpreted as decreasing a temperature, using either a global learning rate (for the entire model) or a learning rate that varies for each parameter. This paper proposes TempBalance, a straightforward yet effective layer-wise learning rate method. TempBalance is based on Heavy-Tailed Self-Regularization (HT-SR) Theory, an approach which characterizes the implicit self-regularization of different layers in trained models. We demonstrate the efficacy of using HT-SR-motivated metrics to guide the scheduling and balancing of temperature across all network layers during model training, resulting in improved performance during testing. We implement TempBalance on CIFAR10, CIFAR100, SVHN, and TinyImageNet datasets using ResNets, VGGs, and WideResNets with various depths and widths. Our results show that TempBalance significantly outperforms ordinary SGD and carefully-tuned spectral norm regularization. We also show that TempBalance outperforms a number of state-of-the-art optimizers and learning rate schedulers.