Fed-ADE: Adaptive Learning Rate for Federated Post-adaptation under Distribution Shift

TL;DR

Fed-ADE introduces an unsupervised, adaptive learning rate method for federated learning that effectively handles non-stationary, heterogeneous data streams by estimating distribution shifts without labels.

Contribution

It proposes a novel unsupervised framework combining uncertainty and representation dynamics for adaptive learning rate adjustment in federated settings under distribution shifts.

Findings

Consistent performance improvements on image and text benchmarks.

Effective handling of label and covariate distribution shifts.

Theoretical guarantees for convergence and regret.

Abstract

Federated learning (FL) in post-deployment settings must adapt to non-stationary data streams across heterogeneous clients without access to ground-truth labels. A major challenge is learning rate selection under client-specific, time-varying distribution shifts, where fixed learning rates often lead to underfitting or divergence. We propose Fed-ADE (Federated Adaptation with Distribution Shift Estimation), an unsupervised federated adaptation framework that leverages lightweight estimators of distribution dynamics. Specifically, Fed-ADE employs uncertainty dynamics estimation to capture changes in predictive uncertainty and representation dynamics estimation to detect covariate-level feature drift, combining them into a per-client, per-timestep adaptive learning rate. We provide theoretical analyses showing that our dynamics estimation approximates the underlying distribution shift and yields dynamic regret and convergence guarantees. Experiments on image and text benchmarks under diverse distribution shifts (label and covariate) demonstrate consistent improvements over strong baselines. These results highlight that distribution shift-aware adaptation enables effective and robust federated post-adaptation under real-world non-stationarity.