On Gossip Algorithms for Machine Learning with Pairwise Objectives

TL;DR

This paper investigates gossip algorithms tailored for pairwise objectives in distributed machine learning, providing a comprehensive theoretical framework and convergence analysis that highlights the influence of network graph properties.

Contribution

It introduces a theoretical framework for gossip algorithms with pairwise objectives, extending beyond average-based functions, and analyzes their convergence properties in networked systems.

Findings

Convergence conditions depend on graph properties.

Refined bounds for convergence rates are established.

The framework applies to similarity learning, ranking, and clustering.

Abstract

In the IoT era, information is more and more frequently picked up by connected smart sensors with increasing, though limited, storage, communication and computation abilities. Whether due to privacy constraints or to the structure of the distributed system, the development of statistical learning methods dedicated to data that are shared over a network is now a major issue. Gossip-based algorithms have been developed for the purpose of solving a wide variety of statistical learning tasks, ranging from data aggregation over sensor networks to decentralized multi-agent optimization. Whereas the vast majority of contributions consider situations where the function to be estimated or optimized is a basic average of individual observations, it is the goal of this article to investigate the case where the latter is of pairwise nature, taking the form of a U -statistic of degree two. Motivated by various problems such as similarity learning, ranking or clustering for instance, we revisit gossip algorithms specifically designed for pairwise objective functions and provide a comprehensive theoretical framework for their convergence. This analysis fills a gap in the literature by establishing conditions under which these methods succeed, and by identifying the graph properties that critically affect their efficiency. In particular, a refined analysis of the convergence upper and lower bounds is performed.