Where is the Testbed for my Federated Learning Research?

TL;DR

The paper introduces CoLExT, a real-world testbed designed to facilitate comprehensive evaluation of federated learning algorithms across diverse devices and metrics, addressing a major gap in FL research infrastructure.

Contribution

It presents CoLExT, a scalable, heterogeneous FL testbed that simplifies experimentation and reveals new insights into FL algorithm performance and implementation issues.

Findings

CoLExT enables easy porting of FL algorithms with minimal overhead.

Initial experiments uncovered new trade-offs and bugs in popular FL algorithms.

The testbed supports real-time metrics collection and visualization.

Abstract

Progressing beyond centralized AI is of paramount importance, yet, distributed AI solutions, in particular various federated learning (FL) algorithms, are often not comprehensively assessed, which prevents the research community from identifying the most promising approaches and practitioners from being convinced that a certain solution is deployment-ready. The largest hurdle towards FL algorithm evaluation is the difficulty of conducting real-world experiments over a variety of FL client devices and different platforms, with different datasets and data distribution, all while assessing various dimensions of algorithm performance, such as inference accuracy, energy consumption, and time to convergence, to name a few. In this paper, we present CoLExT, a real-world testbed for FL research. CoLExT is designed to streamline experimentation with custom FL algorithms in a rich testbed configuration space, with a large number of heterogeneous edge devices, ranging from single-board computers to smartphones, and provides real-time collection and visualization of a variety of metrics through automatic instrumentation. According to our evaluation, porting FL algorithms to CoLExT requires minimal involvement from the developer, and the instrumentation introduces minimal resource usage overhead. Furthermore, through an initial investigation involving popular FL algorithms running on CoLExT, we reveal previously unknown trade-offs, inefficiencies, and programming bugs.