Asynchronous Parallel Incremental Block-Coordinate Descent for Decentralized Machine Learning

TL;DR

This paper introduces an asynchronous parallel incremental block-coordinate descent method for decentralized machine learning, reducing communication costs and improving convergence speed in distributed systems.

Contribution

It proposes a novel API-BCD algorithm for decentralized ML that accelerates convergence and lowers communication costs compared to existing methods.

Findings

API-BCD outperforms state-of-the-art in running time.

API-BCD reduces communication costs significantly.

Convergence properties are rigorously derived.

Abstract

Machine learning (ML) is a key technique for big-data-driven modelling and analysis of massive Internet of Things (IoT) based intelligent and ubiquitous computing. For fast-increasing applications and data amounts, distributed learning is a promising emerging paradigm since it is often impractical or inefficient to share/aggregate data to a centralized location from distinct ones. This paper studies the problem of training an ML model over decentralized systems, where data are distributed over many user devices and the learning algorithm run on-device, with the aim of relaxing the burden at a central entity/server. Although gossip-based approaches have been used for this purpose in different use cases, they suffer from high communication costs, especially when the number of devices is large. To mitigate this, incremental-based methods are proposed. We first introduce incremental block-coordinate descent (I-BCD) for the decentralized ML, which can reduce communication costs at the expense of running time. To accelerate the convergence speed, an asynchronous parallel incremental BCD (API-BCD) method is proposed, where multiple devices/agents are active in an asynchronous fashion. We derive convergence properties for the proposed methods. Simulation results also show that our API-BCD method outperforms state of the art in terms of running time and communication costs.