FedCal: Achieving Local and Global Calibration in Federated Learning via Aggregated Parameterized Scaler

TL;DR

FedCal introduces a novel federated calibration method that uses client-specific scalers and aggregation to improve both local and global model calibration in federated learning, significantly reducing calibration error.

Contribution

The paper proposes FedCal, a new approach that enhances calibration in federated learning by combining local scalers with global aggregation, addressing a gap in calibration research.

Findings

FedCal reduces global calibration error by 47.66% on average.

Existing FL methods have sub-optimal calibration, even with constrained label distribution variance.

FedCal outperforms baseline methods in extensive experiments.

Abstract

Federated learning (FL) enables collaborative machine learning across distributed data owners, but data heterogeneity poses a challenge for model calibration. While prior work focused on improving accuracy for non-iid data, calibration remains under-explored. This study reveals existing FL aggregation approaches lead to sub-optimal calibration, and theoretical analysis shows despite constraining variance in clients' label distributions, global calibration error is still asymptotically lower bounded. To address this, we propose a novel Federated Calibration (FedCal) approach, emphasizing both local and global calibration. It leverages client-specific scalers for local calibration to effectively correct output misalignment without sacrificing prediction accuracy. These scalers are then aggregated via weight averaging to generate a global scaler, minimizing the global calibration error. Extensive experiments demonstrate FedCal significantly outperforms the best-performing baseline, reducing global calibration error by 47.66% on average.