Where is the Testbed for my Federated Learning Research?
Janez Bo\v{z}i\v{c}, Am\^andio R. Faustino, Boris Radovi\v{c}, Marco, Canini, Veljko Pejovi\'c

TL;DR
The paper introduces CoLExT, a real-world testbed designed to facilitate comprehensive evaluation of federated learning algorithms across diverse devices and metrics, addressing a major gap in FL research infrastructure.
Contribution
It presents CoLExT, a scalable, heterogeneous FL testbed that simplifies experimentation and reveals new insights into FL algorithm performance and implementation issues.
Findings
CoLExT enables easy porting of FL algorithms with minimal overhead.
Initial experiments uncovered new trade-offs and bugs in popular FL algorithms.
The testbed supports real-time metrics collection and visualization.
Abstract
Progressing beyond centralized AI is of paramount importance, yet, distributed AI solutions, in particular various federated learning (FL) algorithms, are often not comprehensively assessed, which prevents the research community from identifying the most promising approaches and practitioners from being convinced that a certain solution is deployment-ready. The largest hurdle towards FL algorithm evaluation is the difficulty of conducting real-world experiments over a variety of FL client devices and different platforms, with different datasets and data distribution, all while assessing various dimensions of algorithm performance, such as inference accuracy, energy consumption, and time to convergence, to name a few. In this paper, we present CoLExT, a real-world testbed for FL research. CoLExT is designed to streamline experimentation with custom FL algorithms in a rich testbed…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsPrivacy-Preserving Technologies in Data
