Improving Deep Learning Optimization through Constrained Parameter Regularization

TL;DR

This paper introduces Constrained Parameter Regularization (CPR), a novel regularization method that enforces upper bounds on parameter norms, improving deep learning training by addressing limitations of traditional weight decay.

Contribution

CPR offers a new regularization approach that constrains parameter norms individually, using an augmented Lagrangian method, with minimal runtime overhead and no extensive hyperparameter tuning.

Findings

CPR outperforms weight decay in vision and language tasks.

CPR enhances pre-training and fine-tuning performance.

CPR maintains low computational overhead.

Abstract

Regularization is a critical component in deep learning. The most commonly used approach, weight decay, applies a constant penalty coefficient uniformly across all parameters. This may be overly restrictive for some parameters, while insufficient for others. To address this, we present Constrained Parameter Regularization (CPR) as an alternative to traditional weight decay. Unlike the uniform application of a single penalty, CPR enforces an upper bound on a statistical measure, such as the L2-norm, of individual parameter matrices. Consequently, learning becomes a constraint optimization problem, which we tackle using an adaptation of the augmented Lagrangian method. CPR introduces only a minor runtime overhead and only requires setting an upper bound. We propose simple yet efficient mechanisms for initializing this bound, making CPR rely on no hyperparameter or one, akin to weight decay. Our empirical studies on computer vision and language modeling tasks demonstrate CPR's effectiveness. The results show that CPR can outperform traditional weight decay and increase performance in pre-training and fine-tuning.