Unexpectedly Useful: Convergence Bounds And Real-World Distributed Learning

TL;DR

This paper investigates the practical usefulness of convergence bounds in federated learning, revealing their limitations in prediction but highlighting their potential to identify key clients without data disclosure.

Contribution

The study demonstrates that convergence bounds, despite being loose, can help identify influential clients in federated learning without revealing dataset details.

Findings

Bounds are loose and reflect training loss

Some quantities in bounds identify key clients

Bounds can be exploited to improve distributed learning

Abstract

Convergence bounds are one of the main tools to obtain information on the performance of a distributed machine learning task, before running the task itself. In this work, we perform a set of experiments to assess to which extent, and in which way, such bounds can predict and improve the performance of real-world distributed (namely, federated) learning tasks. We find that, as can be expected given the way they are obtained, bounds are quite loose and their relative magnitude reflects the training rather than the testing loss. More unexpectedly, we find that some of the quantities appearing in the bounds turn out to be very useful to identify the clients that are most likely to contribute to the learning process, without requiring the disclosure of any information about the quality or size of their datasets. This suggests that further research is warranted on the ways -- often counter-intuitive -- in which convergence bounds can be exploited to improve the performance of real-world distributed learning tasks.