ILoRA: Federated Learning with Low-Rank Adaptation for Heterogeneous Client Aggregation

TL;DR

ILoRA introduces a unified federated learning framework with low-rank adaptation that addresses client heterogeneity challenges through orthonormal initialization, rank-aware aggregation, and drift correction, leading to improved accuracy and stability.

Contribution

The paper presents ILoRA, a novel federated learning method combining QR-based initialization, concatenated QR aggregation, and rank-aware optimization to handle heterogeneous client updates effectively.

Findings

Achieves higher accuracy than existing methods.

Provides theoretical convergence guarantees.

Demonstrates robustness across vision and NLP tasks.

Abstract

Federated Learning with Low-Rank Adaptation (LoRA) faces three critical challenges under client heterogeneity: (1) Initialization-Induced Instability due to random initialization misaligning client subspaces; (2) Rank Incompatibility and Aggregation Error when averaging LoRA parameters of different ranks, which biases the global model; and (3) exacerbated Client Drift under Non-IID Data, impairing generalization. To address these challenges, we propose ILoRA, a unified framework that integrates three core innovations: a QR-based orthonormal initialization to ensure all clients start in a coherent subspace; a Concatenated QR Aggregation mechanism that fuses heterogeneous-rank updates via concatenation and decomposition, preserving information while maintaining dimension alignment; and an AdamW optimizer with rank-aware control variates to correct local updates and mitigate client drift. Supported by theoretical convergence guarantees, extensive experiments on vision and NLP benchmarks demonstrate that ILoRA consistently achieves superior accuracy and convergence stability compared to existing federated LoRA methods.