AFL: A Single-Round Analytic Approach for Federated Learning with Pre-trained Models

TL;DR

This paper introduces AFL, an analytic federated learning method that achieves fast, single-round training with invariance to data partitioning, reducing communication overhead and performing well in diverse, non-IID, and large-client scenarios.

Contribution

AFL provides a novel closed-form, gradient-free federated learning approach with single-round aggregation and data partitioning invariance, enhancing efficiency and robustness.

Findings

AFL achieves one-epoch local training eliminating multiple epochs.

The absolute aggregation law enables single-round model aggregation.

AFL performs competitively in highly non-IID and large-client settings.

Abstract

In this paper, we introduce analytic federated learning (AFL), a new training paradigm that brings analytical (i.e., closed-form) solutions to the federated learning (FL) with pre-trained models. Our AFL draws inspiration from analytic learning -- a gradient-free technique that trains neural networks with analytical solutions in one epoch. In the local client training stage, the AFL facilitates a one-epoch training, eliminating the necessity for multi-epoch updates. In the aggregation stage, we derive an absolute aggregation (AA) law. This AA law allows a single-round aggregation, reducing heavy communication overhead and achieving fast convergence by removing the need for multiple aggregation rounds. More importantly, the AFL exhibits a property that \textit{invariance to data partitioning}, meaning that regardless of how the full dataset is distributed among clients, the aggregated result remains identical. This could spawn various potentials, such as data heterogeneity invariance and client-number invariance. We conduct experiments across various FL settings including extremely non-IID ones, and scenarios with a large number of clients (e.g., $\ge 1000$). In all these settings, our AFL constantly performs competitively while existing FL techniques encounter various obstacles. Our codes are available at https://github.com/ZHUANGHP/Analytic-federated-learning.