HiLoRA: Hierarchical Low-Rank Adaptation for Personalized Federated Learning

TL;DR

HiLoRA introduces a hierarchical low-rank adaptation framework for federated learning with Vision Transformers, capturing global, subgroup, and client-specific knowledge to improve personalization and generalization.

Contribution

The paper proposes a novel hierarchical LoRA framework with cross-tier orthogonality and adaptive clustering, addressing latent client structures in federated ViT tuning.

Findings

Consistent improvements in personalization and generalization on CIFAR-100 and DomainNet.

Effective inference of latent client groups via subspace similarity analysis.

Theoretical tier-wise generalization analysis supports the framework's design.

Abstract

Vision Transformers (ViTs) have been widely adopted in vision tasks due to their strong transferability. In Federated Learning (FL), where full fine-tuning is communication heavy, Low-Rank Adaptation (LoRA) provides an efficient and communication-friendly way to adapt ViTs. However, existing LoRA-based federated tuning methods overlook latent client structures in real-world settings, limiting shared representation learning and hindering effective adaptation to unseen clients. To address this, we propose HiLoRA, a hierarchical LoRA framework that places adapters at three levels: root, cluster, and leaf, each designed to capture global, subgroup, and client-specific knowledge, respectively. Through cross-tier orthogonality and cascaded optimization, HiLoRA separates update subspaces and aligns each tier with its residual personalized objective. In particular, we develop a LoRA-Subspace Adaptive Clustering mechanism that infers latent client groups via subspace similarity analysis, thereby facilitating knowledge sharing across structurally aligned clients. Theoretically, we establish a tier-wise generalization analysis that supports HiLoRA's design. Experiments on ViT backbones with CIFAR-100 and DomainNet demonstrate consistent improvements in both personalization and generalization.