A Survey on Split Learning for LLM Fine-Tuning: Models, Systems, and Privacy Optimizations

TL;DR

This survey reviews split learning techniques for fine-tuning large language models, focusing on model optimization, system efficiency, and privacy, to enable resource-constrained and privacy-sensitive applications.

Contribution

It provides the first comprehensive classification and critique of split learning methods for LLM fine-tuning, establishing a structured taxonomy for future research.

Findings

Classifies diverse split learning approaches into a unified framework.

Analyzes key operational components for scalable LLM fine-tuning.

Identifies challenges and future directions in privacy and system efficiency.

Abstract

Fine-tuning unlocks large language models (LLMs) for specialized applications, but its high computational cost often puts it out of reach for resource-constrained organizations. While cloud platforms could provide the needed resources, data privacy concerns make sharing sensitive information with third parties risky. A promising solution is split learning for LLM fine-tuning, which divides the model between clients and a server, allowing collaborative and secure training through exchanged intermediate data, thus enabling resource-constrained participants to adapt LLMs safely. % In light of this, a growing body of literature has emerged to advance this paradigm, introducing varied model methods, system optimizations, and privacy defense-attack techniques for split learning. To bring clarity and direction to the field, a comprehensive survey is needed to classify, compare, and critique these diverse approaches. This paper fills the gap by presenting the first extensive survey dedicated to split learning for LLM fine-tuning. We propose a unified, fine-grained training pipeline to pinpoint key operational components and conduct a systematic review of state-of-the-art work across three core dimensions: model-level optimization, system-level efficiency, and privacy preservation. Through this structured taxonomy, we establish a foundation for advancing scalable, robust, and secure collaborative LLM adaptation.