AutoClip: Adaptive Gradient Clipping for Source Separation Networks

TL;DR

AutoClip introduces an adaptive gradient clipping method that automatically selects optimal thresholds during training, enhancing generalization and smoothing optimization in source separation networks.

Contribution

The paper proposes AutoClip, a novel automatic gradient clipping technique that adapts based on gradient history, improving training stability and performance.

Findings

AutoClip improves generalization in audio source separation.

AutoClip guides training into smoother loss landscape regions.

AutoClip is simple to implement and domain-agnostic.

Abstract

Clipping the gradient is a known approach to improving gradient descent, but requires hand selection of a clipping threshold hyperparameter. We present AutoClip, a simple method for automatically and adaptively choosing a gradient clipping threshold, based on the history of gradient norms observed during training. Experimental results show that applying AutoClip results in improved generalization performance for audio source separation networks. Observation of the training dynamics of a separation network trained with and without AutoClip show that AutoClip guides optimization into smoother parts of the loss landscape. AutoClip is very simple to implement and can be integrated readily into a variety of applications across multiple domains.