Learning Like Humans: Resource-Efficient Federated Fine-Tuning through Cognitive Developmental Stages

TL;DR

This paper presents DevFT, a resource-efficient federated fine-tuning method inspired by human cognitive development, which progressively builds large language models through stages to improve efficiency and performance.

Contribution

DevFT introduces a developmental stage-based federated fine-tuning framework with novel layer grouping and fusion techniques, significantly enhancing efficiency and performance over existing methods.

Findings

Achieves up to 4.59× faster convergence

Reduces communication overhead by 10.67×

Improves performance by 9.07% on average

Abstract

Federated fine-tuning enables Large Language Models (LLMs) to adapt to downstream tasks while preserving data privacy, but its resource-intensive nature limits deployment on edge devices. In this paper, we introduce Developmental Federated Tuning (DevFT), a resource-efficient approach inspired by cognitive development that progressively builds a powerful LLM from a compact foundation. DevFT decomposes the fine-tuning process into developmental stages, each optimizing submodels with increasing parameter capacity. Knowledge from earlier stages transfers to subsequent submodels, providing optimized initialization parameters that prevent convergence to local minima and accelerate training. This paradigm mirrors human learning, gradually constructing comprehensive knowledge structure while refining existing skills. To efficiently build stage-specific submodels, DevFT introduces deconfliction-guided layer grouping and differential-based layer fusion to distill essential information and construct representative layers. Evaluations across multiple benchmarks demonstrate that DevFT significantly outperforms state-of-the-art methods, achieving up to 4.59$\times$ faster convergence, 10.67$\times$ reduction in communication overhead, and 9.07% average performance improvement, while maintaining compatibility with existing approaches.