A Universal Metric of Dataset Similarity for Cross-silo Federated Learning

TL;DR

This paper introduces a novel, privacy-preserving, dataset-agnostic metric for measuring dataset similarity in federated learning, which correlates well with model performance and aids cross-site collaboration.

Contribution

The paper presents the first federated dataset similarity metric that is dataset-agnostic, privacy-preserving, and computationally efficient, addressing limitations of existing metrics.

Findings

The metric correlates robustly with model performance in FL.

It can be computed without data sharing or model training.

The metric is effective across synthetic, benchmark, and medical datasets.

Abstract

Federated Learning is increasingly used in domains such as healthcare to facilitate collaborative model training without data-sharing. However, datasets located in different sites are often non-identically distributed, leading to degradation of model performance in FL. Most existing methods for assessing these distribution shifts are limited by being dataset or task-specific. Moreover, these metrics can only be calculated by exchanging data, a practice restricted in many FL scenarios. To address these challenges, we propose a novel metric for assessing dataset similarity. Our metric exhibits several desirable properties for FL: it is dataset-agnostic, is calculated in a privacy-preserving manner, and is computationally efficient, requiring no model training. In this paper, we first establish a theoretical connection between our metric and training dynamics in FL. Next, we extensively evaluate our metric on a range of datasets including synthetic, benchmark, and medical imaging datasets. We demonstrate that our metric shows a robust and interpretable relationship with model performance and can be calculated in privacy-preserving manner. As the first federated dataset similarity metric, we believe this metric can better facilitate successful collaborations between sites.