A Survey for Federated Learning Evaluations: Goals and Measures

TL;DR

This survey reviews evaluation goals and metrics in federated learning, introduces FedEval as a standardized evaluation platform, and discusses challenges and future directions for FL assessment.

Contribution

It provides a comprehensive overview of evaluation goals, metrics, and introduces FedEval, an open-source platform for standardized FL evaluation.

Findings

Identification of key evaluation goals in FL

Introduction of FedEval platform for standardized assessment

Discussion of challenges and future research directions

Abstract

Evaluation is a systematic approach to assessing how well a system achieves its intended purpose. Federated learning (FL) is a novel paradigm for privacy-preserving machine learning that allows multiple parties to collaboratively train models without sharing sensitive data. However, evaluating FL is challenging due to its interdisciplinary nature and diverse goals, such as utility, efficiency, and security. In this survey, we first review the major evaluation goals adopted in the existing studies and then explore the evaluation metrics used for each goal. We also introduce FedEval, an open-source platform that provides a standardized and comprehensive evaluation framework for FL algorithms in terms of their utility, efficiency, and security. Finally, we discuss several challenges and future research directions for FL evaluation.