Q-LocalAdam: Memory-Efficient Client-Side Adaptive Optimization for Edge Federated Learning

TL;DR

Q-LocalAdam is a memory-efficient adaptive optimizer for edge federated learning that uses distribution-aware quantization to reduce memory overhead without sacrificing accuracy.

Contribution

It introduces a novel distribution-aware 8-bit quantization method for momentum and variance in Adam, enabling significant memory savings on resource-constrained devices.

Findings

Achieves 3.37x optimizer memory reduction with no accuracy loss under moderate heterogeneity.

Significantly improves accuracy under extreme data heterogeneity, e.g., +5.74 percentage points on CIFAR-100.

Distribution-aware quantization outperforms naive uniform quantization, which degrades performance.

Abstract

Federated learning on edge devices must cope with non-IID client data and tight memory budgets. Adaptive optimizers like Adam stabilize training under data heterogeneity but require storing full-precision momentum and variance states, often tripling client memory overhead. This limits deployable model sizes and concurrent federated jobs on resource-constrained devices. We empirically observe that momentum and variance in federated Adam exhibit fundamentally different statistical properties: momentum values are symmetric and bounded, while variance spans eight orders of magnitude with log-normal structure. Motivated by this asymmetry, we propose \textbf{Q-LocalAdam}, which applies distribution-aware 8-bit quantization block-wise linear encoding for momentum and log-space encoding for variance while keeping model parameters in full precision. Across CIFAR-10 and CIFAR-100 under varying data heterogeneity ($\alpha \in \{0.1, 0.5, 1.0, \text{IID}\}$), Q-LocalAdam achieves $3.37\times$ optimizer memory reduction with no accuracy loss under moderate heterogeneity and significant improvements under extreme heterogeneity (e.g., +5.74pp on CIFAR-100, $\alpha=0.1$). Multi-seed validation confirms statistical significance ($p<0.01$). In contrast, naive uniform quantization degrades to random performance, demonstrating that distribution-aware design is essential. Q-LocalAdam enables larger models and more concurrent workloads on memory-constrained edge devices without modifying the federated protocol.