An Information-Theoretic Analysis for Federated Learning under Concept Drift

TL;DR

This paper analyzes how concept drift affects federated learning using information theory and proposes a regularization algorithm to improve long-term performance under various drift patterns, validated through experiments on Raspberry Pi devices.

Contribution

It models concept drift as a Markov chain, introduces the Stationary Generalization Error, and develops a KL divergence and mutual information-based regularization algorithm for federated learning.

Findings

Drift patterns significantly impact FL performance.

The proposed method outperforms existing approaches across drift types.

Experimental validation confirms theoretical insights.

Abstract

Recent studies in federated learning (FL) commonly train models on static datasets. However, real-world data often arrives as streams with shifting distributions, causing performance degradation known as concept drift. This paper analyzes FL performance under concept drift using information theory and proposes an algorithm to mitigate the performance degradation. We model concept drift as a Markov chain and introduce the \emph{Stationary Generalization Error} to assess a model's capability to capture characteristics of future unseen data. Its upper bound is derived using KL divergence and mutual information. We study three drift patterns (periodic, gradual, and random) and their impact on FL performance. Inspired by this, we propose an algorithm that regularizes the empirical risk minimization approach with KL divergence and mutual information, thereby enhancing long-term performance. We also explore the performance-cost tradeoff by identifying a Pareto front. To validate our approach, we build an FL testbed using Raspberry Pi4 devices. Experimental results corroborate with theoretical findings, confirming that drift patterns significantly affect performance. Our method consistently outperforms existing approaches for these three patterns, demonstrating its effectiveness in adapting concept drift in FL.