Asynchronous COMID: the theoretic basis for transmitted data sparsification tricks on Parameter Server

TL;DR

This paper introduces asynchronous COMID, a general algorithm that unifies and proves the convergence of common data sparsification tricks in Parameter Servers, reducing network load without sacrificing convergence speed.

Contribution

It proposes asynchronous COMID, providing a theoretical foundation for data sparsification tricks like FTRL-proximal and L2 norm done at server, and demonstrates their convergence.

Findings

Asynchronous COMID reduces network burden compared to delayed SGD.

The convergence of FTRL-proximal and L2 norm tricks is theoretically proven.

Experimental results confirm no loss in convergence speed or final output.

Abstract

Asynchronous FTRL-proximal and L2 norm done at server are two widely used tricks in Parameters Server which is an implement of delayed SGD. Their commonness is leaving parts of updating computation on server which reduces the burden of network via making transmitted data sparse. But above tricks' convergences are not well-proved. In this paper, based on above commonness, we propose a more general algorithm named as asynchronous COMID and prove its convergence. We prove that asynchronous FTRL-proximal and L2 norm done at server are applications of asynchronous COMID, which demonstrates the convergences of above two tricks. Then, we conduct experiments to verify theoretical results. Experimental results show that compared with delayed SGD on Parameters Server, asynchronous COMID reduces the burden of the network without any harm on the mathematical convergence speed and final output.