Three Mechanisms of Weight Decay Regularization

TL;DR

This paper investigates the underlying mechanisms of weight decay regularization in neural networks, revealing three distinct effects depending on the optimizer and architecture, which enhances understanding and potential improvements.

Contribution

It identifies three different mechanisms through which weight decay regularizes neural networks, challenging traditional interpretations and providing new insights for optimization.

Findings

Weight decay increases effective learning rate in some settings.

It approximately regularizes the input-output Jacobian norm.

It reduces the effective damping in second-order optimizers.

Abstract

Weight decay is one of the standard tricks in the neural network toolbox, but the reasons for its regularization effect are poorly understood, and recent results have cast doubt on the traditional interpretation in terms of $L_2$ regularization. Literal weight decay has been shown to outperform $L_2$ regularization for optimizers for which they differ. We empirically investigate weight decay for three optimization algorithms (SGD, Adam, and K-FAC) and a variety of network architectures. We identify three distinct mechanisms by which weight decay exerts a regularization effect, depending on the particular optimization algorithm and architecture: (1) increasing the effective learning rate, (2) approximately regularizing the input-output Jacobian norm, and (3) reducing the effective damping coefficient for second-order optimization. Our results provide insight into how to improve the regularization of neural networks.