VFLAIR-LLM: A Comprehensive Framework and Benchmark for Split Learning of LLMs

TL;DR

VFLAIR-LLM is a lightweight, extensible framework for split learning of LLMs that enables privacy-preserving inference and fine-tuning in resource-limited environments, with comprehensive benchmarking of attacks and defenses.

Contribution

It introduces VFLAIR-LLM, a novel split learning framework for LLMs supporting multiple tasks and datasets, along with extensive benchmarking of privacy attacks and defenses.

Findings

Benchmarking of 5 attacks and 9 defenses provides insights into privacy risks.

Model partitioning strategies significantly impact privacy and performance.

Guidelines for selecting defense strategies and hyperparameters in split learning.

Abstract

With the advancement of Large Language Models (LLMs), LLM applications have expanded into a growing number of fields. However, users with data privacy concerns face limitations in directly utilizing LLM APIs, while private deployments incur significant computational demands. This creates a substantial challenge in achieving secure LLM adaptation under constrained local resources. To address this issue, collaborative learning methods, such as Split Learning (SL), offer a resource-efficient and privacy-preserving solution for adapting LLMs to private domains. In this study, we introduce VFLAIR-LLM (available at https://github.com/FLAIR-THU/VFLAIR-LLM), an extensible and lightweight split learning framework for LLMs, enabling privacy-preserving LLM inference and fine-tuning in resource-constrained environments. Our library provides two LLM partition settings, supporting three task types and 18 datasets. In addition, we provide standard modules for implementing and evaluating attacks and defenses. We benchmark 5 attacks and 9 defenses under various Split Learning for LLM(SL-LLM) settings, offering concrete insights and recommendations on the choice of model partition configurations, defense strategies, and relevant hyperparameters for real-world applications.