U-Clip: On-Average Unbiased Stochastic Gradient Clipping

TL;DR

U-Clip introduces a novel gradient clipping method that maintains an unbiased estimate of the true gradient by buffering and adding back clipped portions, ensuring convergence in optimization algorithms.

Contribution

It proposes U-Clip, a simple modification to gradient clipping that preserves unbiasedness and improves convergence guarantees in iterative optimization.

Findings

Unbiased gradient estimates achieved with U-Clip.

Effective on CIFAR10 and ImageNet datasets.

Convergence guarantees demonstrated theoretically.

Abstract

U-Clip is a simple amendment to gradient clipping that can be applied to any iterative gradient optimization algorithm. Like regular clipping, U-Clip involves using gradients that are clipped to a prescribed size (e.g. with component wise or norm based clipping) but instead of discarding the clipped portion of the gradient, U-Clip maintains a buffer of these values that is added to the gradients on the next iteration (before clipping). We show that the cumulative bias of the U-Clip updates is bounded by a constant. This implies that the clipped updates are unbiased on average. Convergence follows via a lemma that guarantees convergence with updates $u_i$ as long as $\sum_{i=1}^t (u_i - g_i) = o(t)$ where $g_i$ are the gradients. Extensive experimental exploration is performed on CIFAR10 with further validation given on ImageNet.