Covariances for Free: Exploiting Mean Distributions for Training-free Federated Learning

TL;DR

This paper introduces a training-free federated learning method that leverages class covariance estimations from class means to improve classifier initialization and performance with minimal communication overhead.

Contribution

It proposes a novel covariance-based initialization method using only class means, enhancing federated learning without additional training or high communication costs.

Findings

Improves federated learning performance by 4-26% over mean-only methods.

Achieves competitive results with less communication than second-order methods.

Outperforms federated prompt-tuning approaches in communication efficiency.

Abstract

Using pre-trained models has been found to reduce the effect of data heterogeneity and speed up federated learning algorithms. Recent works have explored training-free methods using first- and second-order statistics to aggregate local client data distributions at the server and achieve high performance without any training. In this work, we propose a training-free method based on an unbiased estimator of class covariance matrices which only uses first-order statistics in the form of class means communicated by clients to the server. We show how these estimated class covariances can be used to initialize the global classifier, thus exploiting the covariances without actually sharing them. We also show that using only within-class covariances results in a better classifier initialization. Our approach improves performance in the range of 4-26% with exactly the same communication cost when compared to methods sharing only class means and achieves performance competitive or superior to methods sharing second-order statistics with dramatically less communication overhead. The proposed method is much more communication-efficient than federated prompt-tuning methods and still outperforms them. Finally, using our method to initialize classifiers and then performing federated fine-tuning or linear probing again yields better performance. Code is available at https://github.com/dipamgoswami/FedCOF.