H2Tune: Federated Foundation Model Fine-Tuning with Hybrid Heterogeneity

TL;DR

H2Tune introduces a novel federated fine-tuning framework for foundation models with hybrid heterogeneity, addressing model and task diversity challenges through matrix decomposition, layer alignment, and knowledge disentanglement, leading to significant accuracy improvements.

Contribution

It proposes H2Tune, a new method for federated foundation model fine-tuning that handles double heterogeneity in model architectures and tasks with innovative alignment and disentanglement techniques.

Findings

Achieves up to 15.4% accuracy improvement over baselines.

Proves convergence rate of O(1/√T).

Effectively aligns heterogeneous models and tasks in federated settings.

Abstract

Different from existing federated fine-tuning (FFT) methods for foundation models, hybrid heterogeneous federated fine-tuning (HHFFT) is an under-explored scenario where clients exhibit double heterogeneity in model architectures and downstream tasks. This hybrid heterogeneity introduces two significant challenges: 1) heterogeneous matrix aggregation, where clients adopt different large-scale foundation models based on their task requirements and resource limitations, leading to dimensional mismatches during LoRA parameter aggregation; and 2) multi-task knowledge interference, where local shared parameters, trained with both task-shared and task-specific knowledge, cannot ensure only task-shared knowledge is transferred between clients. To address these challenges, we propose H2Tune, a federated foundation model fine-tuning with hybrid heterogeneity. Our framework H2Tune consists of three key components: (i) sparsified triple matrix decomposition to align hidden dimensions across clients through constructing rank-consistent middle matrices, with adaptive sparsification based on client resources; (ii) relation-guided matrix layer alignment to handle heterogeneous layer structures and representation capabilities; and (iii) alternating task-knowledge disentanglement mechanism to decouple shared and specific knowledge of local model parameters through alternating optimization. Theoretical analysis proves a convergence rate of O(1/\sqrt{T}). Extensive experiments show our method achieves up to 15.4% accuracy improvement compared to state-of-the-art baselines. Our code is available at https://anonymous.4open.science/r/H2Tune-1407.