FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuning

TL;DR

FedTreeLoRA introduces a tree-structured aggregation method for federated LoRA fine-tuning, effectively balancing statistical and functional heterogeneity across LLM layers to improve personalization and generalization.

Contribution

It proposes a novel tree-based aggregation framework that dynamically aligns client models at different layers, addressing both statistical and functional heterogeneity in federated LoRA training.

Findings

Significantly outperforms existing methods on NLU and NLG benchmarks.

Effectively balances model generalization and personalization.

Demonstrates the importance of layer-wise, hierarchical aggregation in federated learning.

Abstract

Federated Learning (FL) with Low-Rank Adaptation (LoRA) has become a standard for privacy-preserving LLM fine-tuning. However, existing personalized methods predominantly operated under a restrictive Flat-Model Assumption: they addressed client-side \textit{statistical heterogeneity} but treated the model as a monolithic block, ignoring the \textit{functional heterogeneity} across LLM layers. We argue that these two statistical (horizontal) and functional (vertical) dimensions, are \textit{orthogonal in source yet coupled in interaction}, implying that the optimal depth of parameter sharing is functionally dependent on client similarity. To address this, we propose \textbf{FedTreeLoRA}, a framework employing tree-structured aggregation for fine-grained, layer-wise alignment. By dynamically constructing an aggregation hierarchy, FedTreeLoRA allows clients to share broad consensus on shallow `trunks' while progressively specializing on deep `branches'. Experiments on NLU and NLG benchmarks demonstrate that FedTreeLoRA significantly outperforms state-of-the-art methods by effectively reconciling generalization and personalization.