ELSA: Efficient LLM-Centric Split Aggregation for Privacy-Aware Hierarchical Federated Learning over the Network Edge

TL;DR

ELSA is a novel framework that combines split learning and hierarchical federated learning to enable efficient, privacy-aware fine-tuning of large language models at resource-constrained network edges.

Contribution

It introduces a task-agnostic client clustering, adaptive model splitting, and a lightweight communication scheme to improve scalability, privacy, and convergence in edge-based LLM training.

Findings

ELSA outperforms state-of-the-art baselines in NLP tasks.

It achieves better convergence stability and robustness.

The framework reduces communication overhead significantly.

Abstract

Training large language models (LLMs) at the network edge faces fundamental challenges arising from device resource constraints, severe data heterogeneity, and heightened privacy risks. To address these challenges, we propose ELSA (Efficient LLM-centric Split Aggregation), a novel framework that systematically integrates split learning (SL) and hierarchical federated learning (HFL) for distributed LLM fine-tuning over resource-constrained edge networks. ELSA introduces three key innovations. First, it employs a task-agnostic, behavior-aware client clustering mechanism that constructs semantic fingerprints using public probe inputs and symmetric Kullback-Leibler (KL) divergence, augmented by prediction-consistency trust scoring and latency-aware edge assignment to jointly mitigate data heterogeneity, device unreliability, and communication constraints. Second, it employs a resource-aware dynamic model splitting strategy to adaptively partition the LLM into three segments across clients and edge servers, with the cloud used only for adapter aggregation, enabling an effective balance between on-device computation cost and global convergence stability. Third, it incorporates a lightweight communication scheme based on computational sketches combined with semantic subspace orthogonal perturbation (SS-OP) to reduce communication overhead while mitigating privacy leakage during model exchanges across the network. Extensive experiments across diverse NLP tasks demonstrate that ELSA consistently outperforms state-of-the-art baselines in terms of adaptability, convergence behavior, and robustness, establishing a scalable and privacy-aware solution for edge-side LLM fine-tuning under resource constraints.