Forget to Generalize: Iterative Adaptation for Generalization in Federated Learning

TL;DR

This paper introduces Iterative Federated Adaptation (IFA), a novel training paradigm that improves the generalization of federated learning models across heterogeneous, non-IID web data by iteratively reinitializing model parameters.

Contribution

The paper proposes IFA, a new iterative training method that enhances federated learning generalization by selectively reinitializing model parameters to escape local minima.

Findings

21.5% average accuracy improvement across datasets

Effective on top of existing federated algorithms

Significantly improves performance on non-IID data

Abstract

The Web is naturally heterogeneous with user devices, geographic regions, browsing patterns, and contexts all leading to highly diverse, unique datasets. Federated Learning (FL) is an important paradigm for the Web because it enables privacy-preserving, collaborative machine learning across diverse user devices, web services and clients without needing to centralize sensitive data. However, its performance degrades severely under non-IID client distributions that is prevalent in real-world web systems. In this work, we propose a new training paradigm - Iterative Federated Adaptation (IFA) - that enhances generalization in heterogeneous federated settings through generation-wise forget and evolve strategy. Specifically, we divide training into multiple generations and, at the end of each, select a fraction of model parameters (a) randomly or (b) from the later layers of the model and reinitialize them. This iterative forget and evolve schedule allows the model to escape local minima and preserve globally relevant representations. Extensive experiments on CIFAR-10, MIT-Indoors, and Stanford Dogs datasets show that the proposed approach improves global accuracy, especially when the data cross clients are Non-IID. This method can be implemented on top any federated algorithm to improve its generalization performance. We observe an average of 21.5%improvement across datasets. This work advances the vision of scalable, privacy-preserving intelligence for real-world heterogeneous and distributed web systems.