Grad Queue : A probabilistic framework to reinforce sparse gradients

TL;DR

This paper introduces Grad Queue, a probabilistic framework that enhances sparse gradients in large batch training by reinforcing rare gradient components, clustering data to reduce conflicts, and dynamically adjusting the gradient queue to improve convergence and performance.

Contribution

It presents a novel probabilistic mechanism with gradient reinforcement, clustering, and dynamic queue management to improve large batch training efficiency and accuracy.

Findings

Superior performance on CIFAR10, MNIST, and Reuters datasets.

Restores intra-mini-batch diversity and widens the optimal batch boundary.

Effective in deepening minima convergence.

Abstract

Informative gradients are often lost in large batch updates. We propose a robust mechanism to reinforce the sparse components within a random batch of data points. A finite queue of online gradients is used to determine their expected instantaneous statistics. We propose a function to measure the scarcity of incoming gradients using these statistics and establish the theoretical ground of this mechanism. To minimize conflicting components within large mini-batches, samples are grouped with aligned objectives by clustering based on inherent feature space. Sparsity is measured for each centroid and weighted accordingly. A strong intuitive criterion to squeeze out redundant information from each cluster is the backbone of the system. It makes rare information indifferent to aggressive momentum also exhibits superior performance with larger mini-batch horizon. The effective length of the queue kept variable to follow the local loss pattern. The contribution of our method is to restore intra-mini-batch diversity at the same time widening the optimal batch boundary. Both of these collectively drive it deeper towards the minima. Our method has shown superior performance for CIFAR10, MNIST, and Reuters News category dataset compared to mini-batch gradient descent.