Regularized Top-$k$: A Bayesian Framework for Gradient Sparsification

TL;DR

This paper introduces a Bayesian regularization approach to gradient sparsification called RegTop-k, which improves convergence and performance in distributed training by controlling error accumulation.

Contribution

It develops a novel Bayesian-based sparsification scheme that regularizes Top-k, enhancing convergence and accuracy at high compression ratios in distributed learning.

Findings

RegTop-k converges to the global optimum faster than Top-k.

RegTop-k outperforms Top-k in distributed training of ResNet-18 on CIFAR-10.

Higher compression ratios benefit from the proposed regularization.

Abstract

Error accumulation is effective for gradient sparsification in distributed settings: initially-unselected gradient entries are eventually selected as their accumulated error exceeds a certain level. The accumulation essentially behaves as a scaling of the learning rate for the selected entries. Although this property prevents the slow-down of lateral movements in distributed gradient descent, it can deteriorate convergence in some settings. This work proposes a novel sparsification scheme that controls the learning rate scaling of error accumulation. The development of this scheme follows two major steps: first, gradient sparsification is formulated as an inverse probability (inference) problem, and the Bayesian optimal sparsification mask is derived as a maximum-a-posteriori estimator. Using the prior distribution inherited from Top-k, we derive a new sparsification algorithm which can be interpreted as a regularized form of Top-k. We call this algorithm regularized Top-k (RegTop-k). It utilizes past aggregated gradients to evaluate posterior statistics of the next aggregation. It then prioritizes the local accumulated gradient entries based on these posterior statistics. We validate our derivation through various numerical experiments. In distributed linear regression, it is observed that while Top-k remains at a fixed distance from the global optimum, RegTop-k converges to the global optimum at significantly higher compression ratios. We further demonstrate the generalization of this observation by employing RegTop-k in distributed training of ResNet-18 on CIFAR-10, as well as fine-tuning of multiple computer vision models on the ImageNette dataset. Our numerical results confirm that as the compression ratio increases, RegTop-k sparsification noticeably outperforms Top-k.